.. highlight:: shell-session

.. _chapter-sage-trac:

====================
The Sage Trac Server
====================

Sometimes you will only want to work on local changes to Sage, for
your own private needs.  However, typically it is beneficial to
share code and ideas with others; the manner in which the
`Sage project <https://www.sagemath.org>`_ does this (as well as fixing
bugs and upgrading components) is in a very collaborative and
public setting on `the Sage Trac server <https://trac.sagemath.org>`_
(the Sage bug and enhancement tracker).

The purpose of the Sage trac server is to

1. Provide a place for discussion on issues and store a permanent
   record.

2. Provide a repository of source code and all proposed changes.

3. Link these two together.

There is also a :trac:`wiki <wiki>` for more general
organizational web pages, like Sage development workshops.

Thus if you find a bug in Sage, if you have new code to submit, want
to review new code already written but not yet included in Sage, or if
you have corrections for the documentation, you should post on the
trac server. Items on the server are called *tickets*, and anyone may
search or browse the tickets. For a list of recent changes, just visit
the :trac:`Sage trac timeline <timeline>`.

.. _section-trac-account:

Obtaining an Account
====================

If you do not have an account on GitHub yet, choose a user name and
`create an account <https://github.com/join>`_.

Using your GitHub account, you can log in to:

- `the Sage trac server <https://trac.sagemath.org>`_, so that you can
  open tickets and participate in discussions on existing tickets.

  On the Sage trac server, click the link "GitHub Login" in the top
  right corner and follow the prompts.

  Within the Sage trac server, your GitHub user name will be prefixed
  with the letters ``gh-`` (which stand for "GitHub").

- GitLab, where the mirror repository `sagemath/sage
  <https://gitlab.com/sagemath/sage>`_ accepts Merge Requests.

  In GitLab, click the button "Sign in / Register" in the top right
  corner, and then use the button "Sign in with GitHub" and follow the
  prompts.

Users with legacy sage-trac accounts (account names not starting with
"gh-") should use the Login link. Do not to use GitHub login, as it
will be treated as a separate user from their original account (unless
you actively prefer to switch).


.. _trac-bug-report:

Reporting Bugs
==============

If you think you have found a bug in Sage, here is the procedure:

- Search through our Google groups for postings related to your possible bug (it
  may have been fixed/reported already):

  * ``sage-devel``: `<https://groups.google.com/group/sage-devel>`_
  * ``sage-support``: `<https://groups.google.com/group/sage-support>`_

  Similarly, you can search :ref:`chapter-sage-trac` to see if anyone else has
  opened a ticket about your bug.

- If you do not find anything, and you are not sure that you have found a bug,
  ask about it on `sage-devel <https://groups.google.com/group/sage-devel>`_. A
  bug report should contain:

  - An explicit and **reproducible example** illustrating your bug (and/or the
    steps required to reproduce the buggy behavior).

  - The **version** of Sage you run, as well as the version of the optional
    packages that may be involved in the bug.

  - Describe your **operating system** as accurately as you can and your
    architecture (32-bit, 64-bit, ...).

- You might be asked to open a new ticket. In this case, follow the
  :ref:`section-trac-new-ticket`.

Thank you in advance for reporting bugs to improve Sage in the future!

.. _section-trac-new-ticket:

Guidelines for Opening Tickets
==============================

In addition to bug reports (see :ref:`trac-bug-report`), you should also open a
ticket if you have some new code that makes Sage a better tool. If you have a
feature request, start a discussion on ``sage-devel`` first, and then if there
seems to be general agreement that you have a good idea, open a ticket
describing the idea.

- Do you already have a **trac account**? If not, :ref:`click here
  <section-trac-account>`.

**Before** opening a new ticket, consider the following points:

- Make sure that nobody else has opened a ticket about the same or closely
  related issue.

- It is much better to open several specific tickets than one that
  is very broad. Indeed, a single ticket which deals with lots of
  different issues can be quite problematic, and should be avoided.

- Be precise: If foo does not work on OS X but is fine on Linux,
  mention that in the title. Use the keyword option so that
  searches will pick up the issue.

- The problem described in the ticket must be solvable. For
  example, it would be silly to open a ticket whose purpose was
  "Make Sage the best mathematical software in the world". There is
  no metric to measure this properly and it is highly subjective.

- For bug reports: the ticket's description should contain the information
  described at :ref:`trac-bug-report`.

- If appropriate, provide URLs to background information or sage-devel
  conversation relevant to the problem you are reporting.

**When creating** the ticket, you may find useful to read
:ref:`section-trac-fields`.

Unless you know what you are doing, leave the milestone field to its default
value.

.. _section-trac-fields:

The Ticket Fields
=================

When you open a new ticket or change an existing ticket, you will find a variety
of fields that can be changed. Here is a comprehensive overview (for the
'status' field, see :ref:`section-trac-ticket-status`):

* **Reported by:** The trac account name of whoever created the
  ticket. Cannot be changed.

* **Owned by:** Trac account name of owner, by default the person in charge of
  the Component (see below). Generally not used in the Sage trac.

* **Type:** One of ``enhancement`` (e.g. a new feature), ``defect`` (e.g. a bug
  fix), or ``task`` (rarely used).

* **Priority:** The priority of the ticket. Keep in mind that the
  "blocker" label should be used very sparingly.

* **Milestone:** Milestones are usually goals to be met while working
  toward a release. In Sage’s trac, we use milestones instead of
  releases. Each ticket must have a milestone assigned. If you are
  unsure, assign it to the current milestone.

* **Component:** A list of components of Sage, pick one that most
  closely matches the ticket.

* **Keywords:** List of keywords. Fill in any keywords that you think
  will make your ticket easier to find. Tickets that have been worked
  on at Sage days ``NN`` (some number) ofter have ``sdNN`` as keyword.

* **Cc:** List of trac user names to Cc (send emails for changes on
  the ticket). Note that users that enter a comment are automatically
  substcribed to email updates and don't need to be listed under Cc.

* **Merged in:** The Sage release where the ticket was merged in. Only
  changed by the release manager.

* **Authors:** Real name of the ticket author(s). Set this field only if you
  intend to provide code.

* **Reviewers:** Real name of the ticket reviewer(s).

* **Report Upstream:** If the ticket is a bug in an upstream component
  of Sage, this field is used to summarize the communication with the
  upstream developers.

* **Work issues:** Issues that need to be resolved before the ticket
  can leave the "needs work" status.

* **Branch:** The Git branch containing the ticket's code (see
  :ref:`section-walkthrough-branch`). It is displayed in green color,
  unless there is a conflict between the branch and the latest beta
  release (red color). In this case, the branch should be merged or
  rebased.

* **Dependencies:** Does the ticket depend on another ticket?
  Sometimes, a ticket requires that another ticket be applied
  first. If this is the case, put the dependencies as a
  comma-separated list (``#1234, #5678``) into the "Dependencies:"
  field.

* **Stopgaps:** See :ref:`section-trac-stopgaps`.

.. _section-trac-ticket-status:

The Status of a Ticket
======================

The status of a ticket appears right next to its number, at the top-left corner
of its page. It indicates who has to work on it.

- **new** -- the ticket has only been created (or the author forgot to change
  the status to something else).

  If you intend to work on the code yourself, put your name in the Authors
  field, or leave a comment to say so. It could avoid having two persons doing
  the same job.

- **needs_review** -- the code is ready to be peer-reviewed. If the code is not
  yours, then you can review it. See :ref:`chapter-review`.

- **needs_work** -- something needs to be changed in the code. The reason should
  appear in the comments.

- **needs_info** -- somebody has to answer a question before anything else can
  happen. It should be clear from the comments.

- **positive_review** -- the ticket has been reviewed, and the release manager
  will close it.

The status of a ticket can be changed using a form at the bottom of the ticket's
page. Leave a comment explaining your reasons whenever you change it.

.. _section-trac-stopgaps:

Stopgaps
========

When Sage returns wrong results, two tickets should be opened:

- A main ticket with all available details.
- A "stopgap" ticket (e.g. :trac:`12699`)

This second ticket does not fix the problem but adds a warning that will be
printed whenever anyone uses the relevant code. This, until the problem is
finally fixed.

To produce the warning message, use code like the following:

.. CODE-BLOCK:: python

    from sage.misc.stopgap import stopgap
    stopgap("This code contains bugs and may be mathematically unreliable.",
        TICKET_NUM)

Replace ``TICKET_NUM`` by the ticket number for the main ticket. On the main
trac ticket, enter the ticket number for the stopgap ticket in the "Stopgaps"
field (see :ref:`section-trac-fields`). Stopgap tickets should be marked as
blockers.

.. NOTE::

    If mathematically valid code causes Sage to raise an error or
    crash, for example, there is no need for a stopgap.  Rather,
    stopgaps are to warn users that they may be using buggy code; if
    Sage crashes, this is not an issue.


Working on Tickets
==================

If you manage to fix a bug or enhance Sage you are our hero. See
:ref:`chapter-walkthrough` for making changes to the Sage source
code, uploading them to the Sage trac server, and finally putting your
new branch on the trac ticket.

.. image:: ticket_badges.png

After pushing a branch to a ticket, the ticket will show badges
linking to results of automated tests that run on the patchbot and
other tests that run on GitHub Actions.

* The Patch buildbot will automatically test your ticket. See :trac:`wiki/patchbot`
  for more information about its features and limitations. Make sure that you
  look at the log, especially if the patch buildbot did not give you
  the green blob.

* A `linting workflow
  <https://github.com/sagemath/sage/blob/develop/.github/workflows/lint.yml>`_
  runs on all pushes to a branch on Trac. It checks that the code of
  the current branch adheres to the style guidelines using
  :ref:`section-tools-pycodestyle` (in the ``pycodestyle-minimal``
  configuration) and :ref:`section-tools-relint`.

  In order to see details when it fails, you can click on the badge
  and then select the most recent workflow run.

* The `incremental build and test workflow
  <https://github.com/sagemath/sage/blob/develop/.github/workflows/build.yml>`_
  on GitHub Actions builds Sage for the current branch (incrementally
  on top of an installation of the ``develop`` branch) and runs the
  test.  Note that in contrast to the patchbot, the ticket branch is
  not merged into the current beta version.

  Details are again available by clicking on the badge.

  The automatic workflow runs on a container based on
  ``ubuntu-focal-standard``.  To request a run of the workflow on a
  different platform, you can issue a `workflow_dispatch
  <https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow#running-a-workflow>`_.
  You can select any of the platforms for which a `prebuilt container
  image
  <https://github.com/orgs/sagemath/packages?tab=packages&q=with-targets-optional>`_
  exists.

* The `build documentation workflow
  <https://github.com/sagemath/sage/blob/develop/.github/workflows/doc-build.yml>`_
  on GitHub Actions builds the HTML documentation for the current
  branch.

  If you click on the badge, you get the HTML output of the successful
  run. The idea is to use this to easily inspect changes to the
  documentation without the need to locally rebuild the docs
  yourself. If the doc build fails, you can go to `the Actions tab of
  sagemath/sagetrac-mirror repo
  <https://github.com/sagemath/sagetrac-mirror/actions/workflows/doc-build.yml>`_
  and choose the particular branch to see what went wrong.


The following are some other relevant issues:

* Every bug fixed should result in a doctest.

* This is not an issue with defects, but there are many enhancements
  possible for Sage and too few developers to implement all the good
  ideas. The trac server is useful for keeping ideas in a central
  place because in the Google groups they tend to get lost once they
  drop off the first page.

* If you are a developer, be nice and try to solve a stale/old ticket
  every once in a while.

* Some people regularly do triage. In this context, this means that we
  look at new bugs and classify them according to our perceived
  priority. It is very likely that different people will see
  priorities of bugs very differently from us, so please let us know
  if you see a problem with specific tickets.

Reviewing and Closing Tickets
=============================

Tickets can be closed when they have positive review or for other reasons. To
learn how to review, please see :ref:`chapter-review`.

Only the Sage release manager will close tickets. Most likely, this is
not you nor will your trac account have the necessary permissions. If
you feel strongly that a ticket should be closed or deleted, then
change the status of the ticket to *needs review* and change the
milestone to *sage-duplicate/invalid/wontfix*. You should also
comment on the ticket, explaining why it should be closed. If another
developer agrees, he sets the ticket to *positive review*.

A related issue is re-opening tickets. You should refrain from
re-opening a ticket that is already closed. Instead, open a new ticket
and provide a link in the description to the old ticket.

Reasons to Invalidate Tickets
=============================

**One Issue Per Ticket**: A ticket must cover only one issue
and should not be a laundry list of unrelated issues. If a ticket
covers more than one issue, we cannot close it and while some of
the patches have been applied to a given release, the ticket would
remain in limbo.

**No Patch Bombs**: Code that goes into Sage is peer-reviewed. If
you show up with an 80,000 lines of code bundle that completely
rips out a subsystem and replaces it with something else, you can
imagine that the review process will be a little tedious. These
huge patch bombs are problematic for several reasons and we prefer
small, gradual changes that are easy to review and apply. This is
not always possible (e.g. coercion rewrite), but it is still highly
recommended that you avoid this style of development unless there
is no way around it.

**Sage Specific**: Sage's philosophy is that we ship everything
(or close to it) in one source tarball to make debugging possible.
You can imagine the combinatorial explosion we would have to deal
with if you replaced only ten components of Sage with external
packages. Once you start replacing some of the more essential
components of Sage that are commonly packaged (e.g. Pari, GAP,
lisp, gmp), it is no longer a problem that belongs in our tracker.
If your distribution's Pari package is buggy for example, file a
bug report with them. We are usually willing and able to solve
the problem, but there are no guarantees that we will help you
out. Looking at the open number of tickets that are Sage specific,
you hopefully will understand why.

**No Support Discussions**: The trac installation is not meant to
be a system to track down problems when using Sage. Tickets should
be clearly a bug and not "I tried to do X and I couldn't get it to
work. How do I do this?" That is usually not a bug in Sage and it
is likely that ``sage-support`` can answer that question for you. If
it turns out that you did hit a bug, somebody will open a concise
and to-the-point ticket.

**Solution Must Be Achievable**: Tickets must be achievable. Many
times, tickets that fall into this category usually ran afoul to
some of the other rules listed above. An example would be to
"Make Sage the best CAS in the world". There is no metric to
measure this properly and it is highly subjective.

The Release Process
===================

The Sage Release Manager uses the following procedure to make releases, as of
2022.

**Beta Release Stage**: For preparing a new beta release or the first release
candidate, all positively reviewed tickets with the forthcoming release
milestone are considered. Tickets that have unmerged dependencies are ignored.
The Release Manager merges tickets in batches of 10 to 20 tickets, taking the
ticket priority into account. If a merge conflict of a ticket to the Release
Manager's branch occurs, the ticket is set back to "needs work" status by the
Release Manager, and the list of the tickets already merged to the Release
Manager's branch is posted. The author of the ticket needs to identify
conflicting tickets in the list, make merge commits and declare them as
dependencies, before setting back to "positive review" status. Each batch of
merged tickets then undergoes integration testing. If problems are detected, a
ticket will be set back to "needs work" status and unmerged. When a batch of
tickets is ready, the Release Manager closes these tickets and proceeds to the
next batch. After a few batches, a new beta release is tagged, pushed to the
``develop`` branch on the main git repository, and announced on
``sage-release``.

**Release Candidate Stage**: After the first release candidate has been made,
the project is in the release candidate stage, and a modified procedure is
used. Now **only tickets with a priority set to "blocker" are considered**.
Tickets with all other priorities, including "critical", are ignored. Hence if
a ticket is important enough to merit inclusion in this stage, it should be set
to "blocker".

**Blocker Tickets**: The goal of the release process is to make a stable
release of high quality. Be aware that there is a risk/benefit trade-off in
merging a ticket. The benefit of merging a ticket is the improvement that the
ticket brings, such as fixing a bug. However, any code change has a risk of
introducing unforeseen new problems and thus delaying the release: If a new
issue triggers another release candidate, it delays the release by 1-2 weeks.
Hence developers should use "blocker" priority sparingly and should indicate
the rationale on the ticket. Though there is no one fixed rule or authority
that determines what is appropriate for "blocker" status,

- Tickets introducing new features are usually not blockers -- unless perhaps
  they round out a set of features that were the focus of development of this
  release cycle.

- Tickets that make big changes to the code, for example refactoring tickets,
  are usually not blockers.

**Final Release**: If there is no blocker ticket for the last release
candidate, the Release Manager turns it to the final release. It is tagged with
the release milestone, and announced on ``sage-release``.
