= Repository Editing and Conversion With Reposurgeon
:doctype: book
:author: Eric S. Raymond
:docinfo: shared
:toc: left
:toclevels: 2
:!webfonts:
:sectlinks:
:sectanchors:
:sectnums:
:dtanchors:

// Note: This master is full of inclusions that are the output of
// reposurgeon embedded-help commands.  It's done this way so that,
// as often as possible, there is only a single authoritative text
// on what a command does.  The exceptions are places where this
// manual wants or needs to use more elaborate asciidoc markup than
// we want to put in an embedded help entry.
//
// Each inclusion is merged with either TOPIC or COMMAND.  TOPIC
// is used by the script that generates the browsable form of
// the documentation. COMMAND is not currently used, it's there so
// all inclusions are marked and it's easier to check that there
// are no missing markers.


[[howto]]
== How to use this manual

This is the long-form manual for the reposurgeon tool suite.

Everybody should read the <<introduction>> (immediately after this
section) to be sure reposurgeon is the tool you actually want.

Then read the <<quickstart>> section (immediately following the
Introduction) to get a feeling for the syntax and style of reposurgeon
commands.

Assuming you're trying to do a repository lift, you should probably
continue by reading <<conversion>>.  It is not necessary to read the
entire main body of the manual after that (the command reference)
straight through, but keep it handy for when you need to learn more
about a particular command or group of commands.

If you're doing something more unusual than a conversion, you'll
probably have to read through the command reference until
you discover the commands you need.

Help is available within reposurgeon using the "<<help_cmd>>" command.
Type "help" alone for a list of help topics.

If you find an error or omission in this document, you can suggest an
edit https://gitlab.com/esr/reposurgeon/-/blob/master/repository-editing.adoc[here].

[[introduction]]
== Introduction

The purpose of reposurgeon is to enable risky operations that VCSes
(version-control systems) don't want to let you do, such as (a)
editing past comments and metadata, (b) excising commits, (c)
coalescing and splitting commits, (d) removing files and subtrees from
repo history, (e) merging or grafting two or more repos, and (f)
cutting a repo in two by cutting a parent-child link, preserving the
branch structure of both child repos.

A major use of reposurgeon is to assist a human operator to perform
higher-quality conversions among version-control systems than can be
achieved with fully automated converters. Another application is when
code needs to be removed from a repository for legal or policy reasons.

Fully supported systems (those for which reposurgeon can both read and
write repositories and the support has been tested) include git, hg,
bzr, brz, fossil, darcs, RCS, and SRC. There is tested read-side support for
SCCS, CVS and svn. There is untested read-write support for BitKeeper
and untested read support for Monotone. See <<support> for more details.

Writing to the file-oriented systems RCS and SRC has some serious
limitations because those systems cannot represent all the metadata in
a git-fast-export stream. They require rcs-fast-import as a back end;
consult that tool's documentation for details and partial workarounds.

SCCS, SVN, and CVS are supported for read only, not write. For CVS,
reposurgeon must be run from within a repository directory (that is, a
tree of CVS masters; a CVSROOT is not required). When reading from a
CVS top-level directory each module becomes a subdirectory in the
reposurgeon representation of the change history. It is also possible
to read a repository from within a CVS module subdirectory and lift
that individual module.

Perforce (p4) is not directly supported, but the auxiliary program
repotool(1) can mirror a p4 repository as a local git repository.

Note that reposurgeon is a sharp enough tool to cut you.  It never
modifies a repository in place, and it takes care not to ever write a
repository in an actually inconsistent state, and will terminate with
an error message rather than proceed when its internal data structures
are confused.  However, there are lots of things you can do with it -
like altering stored commit timestamps so they no longer match the
commit sequence - that are likely to cause havoc after you're done.
Proceed with caution and check your work.

Also note that, if your DVCS does the usual thing of making commit IDs
a cryptographic hash of content and parent links, editing a
publicly-accessible repository with this tool would be a bad idea. All
of the surgical operations in reposurgeon will modify the hash chains.

Please also see the notes on system-specific issues under
<<limitations>>.

[[quickstart]]
== Quick start

`reposurgeon` accepts commands as command line arguments. For example:

--------------------------
$ reposurgeon "read projectdir" lint
--------------------------

This will read repository into memory from `projectdir` and check it
for problems. The quotes are required so the read command and its
argument will be presented to the command interpreter as a single
string.

You can use "help" and "help <command>" to learn more about
`reposurgeon` commands, or just read on.

Typically you put these commands in a script that you evolve by
experimenting until you got a conversion that suits your tastes and
needs. `"script"` command allows to read such command list from a file.

--------------------------
$ reposurgeon "do projectscript.lift"
--------------------------

As a motivating example, the script below does a Subversion-to-Git
lift on a repository named _project_ just under the current directory.

[source,bash]
--------
# Load the project into main memory
# Warning: this command is slow because Subversion is slow.
read project

# Transcode Latin-1 characters in comments and attributions
# to UTF-8.
=C transcode latin1

# Check for and report glitches such as timestamp collisions,
# ill-formed committer/author IDs, multiple roots, etc.
lint

# Map Subversion usernames to Git-style user IDs.  In a real
# conversion you'd probably have a lot more of these and you'd
# probably read them in from a separate file, not a heredoc.
authors read <<EOF
fred = Fred Foonly <fred@foonly.net> America/Chicago
jrh = J. Random Hacker <jrh@random.org> America/Los_Angeles
esr = Eric S. Raymond <esr@thyrsus.com> America/New_York
julien = Julien '_FrnchFrgg_' RIVAUD <julien@frnchfrgg.pw> Europe/Paris
db48x = Daniel Brooks <db48x@db48x.net> America/Los_Angeles
ecree = Edward Cree <ecree@solarflare.com> Europe/London
ian = Ian Bruene <ianbruene@gmail.com>
sunshine = Eric Sunshine <sunshine@sunshineco.com>
EOF

# Massage comments into Git-like form (with a topic sentence and a
# spacer line after it if there is following running text). Only
# done when the first line is syntactically recognizable as a whole
# sentence.
gitify

# Tags with the name prefix emptycommit were branch-creation commits
# in Subversion. Usually there's nothing interesting in the comment
# text, but you'll want to browse them and check.  These commands
# save one such tag and delete the rest
tag move emptycommit-23 noteworthy
delete tag /emptycommit/

# Delete remnant .cvsignore files from a past life as CVS.
delete path /.cvsignore$/

# Rename and translate the syntax of ignore files
ignores --defaults --translate

# Often, your Subversion repository was a CVS repository in a past
# life. CVS creates tags named with the suffix "-root" to mark branch
# points for its internal housekeeping, and cv2svn blindly copied them
# even though the Subversion tools don't need that marker. This
# clutter can be tossed.
tag delete /-root$/

# This command illustrates how to use msgin to modify the comment
# text of a commit. In this test we're patching a Subversion revision
# reference because we're going to want to reference-lift it later.
# But this capability could also be used, for example, to add an
# update note to a commit comment that turned out to be incorrect.
msgin <<EOF
Legacy-ID: 23
Check-Text: Referring back to r2.

Referring back to [[SVN:2]].
EOF

# Ensure that all author dates are unique, bumping where necessary. Then
# change cookies like [[SVN:2]] into action stamps that are independent
# of the VCS in use. A typical action stamp looks like this:
# "jrh@random.org!2006-08-14T02:34:56Z".
timequake
stampify

# Sometimes it's useful to drop files from the repo that should
# never have been checked in. 1..$ - selects all commits.
1..$ delete path /documents/.*.pdf/

# Process GNU Changelogs to get better attributions for changesets.
# When a commit was derived from a patch and checked in by someone
# other than its author this can often correct the commit attribution.
changelogs

# It's good practice to add a tag marking the point of conversion.
create tag cutover-git
msgin <<EOF
Tag-Name: cutover-git
Tagger: J. Random Hacker <jrh@random.org> America/Los_Angeles

This tag marks the last Subversion commit before the move to Git.
EOF

# We want to write a Git repository
prefer git

# Do it
rebuild project-git
--------

You can find more usage examples in <<tipsntricks>>.

[[support]]
== Supported Version Control Systems

The design objective of reposurgeon is to support history
interconversion and repository surgery on every VCS in use on Unix
systems.  This is a large, messy domain. Only a few systems have
top-tier support with good coverage in the reposurgeon test suite;
others are theoretically supported but only lightly tested and will
probably require reposurgeon development work when they are first
encountered in the wild.

=== Git

The internal representation reposurgeon uses for repository states
losslessly captures Git histories. Both import and export are fully
supported.

Git has top-tier support with excellent testing.

=== CVS

Reposurgeon can import, but not export, CVS repositories.  You must have
a copy of the repository master directory, and run reposurgeon pointed
at that directory or one of its module subdirectories; a checkout
directory won't do.

Beware that the quality of CVS lifts may be poor, with individual
lifts requiring serious hand-hacking. This is due to inherent problems
with CVS's file-oriented model and the fragility of its repository
structures; reposurgeon cannot entirely compensate for these flaws.
See <<CVS>> for discussion and workarounds.

CVS has top-tier support with good testing. It has been used for many
CVS-to-Git conversions.

It is extremely unlikely that export to CVS will ever be implemented. (This
means: somebody would have to pay the maintainer real money to make it
happen.)

=== Subversion

Reposurgeon can import Subversion dumpfiles or repositories. If you want
it to read a repository, you must run it within the top-level
directory of the repository itself, *not* in a checkout directory made
from the repository.

Unlike CVS, Subversion repositories have real changesets, and the work
in them can effectively always be mapped to equivalent commits in a
more modern system.  The parent-child relationships among commits will
also translate cleanly.

Operator errors, prior conversion with tools like cvs2svn, and
gatewaying with git-svn can leave cruft in a Subversion history
that needs to be cleaned out.  See <<subversion>> for discussion
and workarounds.

Subversion has top-tier support with excellent testing. It has been
used for many, many Subversion lifts to Git; this is the best-tested
of any of the conversion paths.

It is not likely that export to Subversion will ever be
implemented. (This means: somebody would have to pay the maintainer
real money to make it happen, but there is a plausible future in which
that happens.)

=== Mercurial

Mercurial (hg): reposurgeon can import and export Mercurial
repositories; see <<mercurial>> for details. Reposurgeon has been used
for successful history conversions from Mercurial to Git.

Mercurial has top-tier support and good test coverage.

See <<mercurial>> for more details.

=== Bazaar/Breezy

Bazaar and Breezy (bzr, brz): These are workalikes. Import and export
have only basic test coverage, but reposurgeon has been successfully
used for a major Bazaar-to-Git conversion (the history of Emacs).

Some Bazaar extensions - per-commit properties and multiple authors -
are directly supported in reposurgeon. On the other hand, nothing in
Bazaar corresponds to a Git annotated tag, so those would be lost in a
Git-to-Bazaar conversion. It is not believed that any such thing has
ever been attempted.

Bazaar has top-tier support, and Breezy is expected to replicate its
behavior closely enough to inherit that. Test coverage is only basic,
however.

=== Second-tier systems

A VCS is "second tier" if import capability is checked in the
reposurgeon test suite, but it hasn't had known use for a
conversion in the wild.

RCS: reposurgeon will read an RCS collection.  It uses
http://www.catb.org/~esr/src[SRC] to get and export a fast-import
stream to reposurgeon's importer. This importer does not attempt to
make changesets out of cliques of per-file commits; you'll have to hp
this yourself using reposurgeon's coalesce command.

SRC: reposurgeon can read an SRC collection. The caveats for RCS apply.

Writing to the file-oriented SRC and RCS systems can be done via
rcs-fast-import(1).  It has some serious limitations because RCS
cannot represent all the metadata in a git-fast-export stream. Consult
that tool's documentation for details and partial workarounds.

SCCS: reposurgeon will read an SCCS collection. It uses
http://www.catb.org/esr/src/[SRC] to get and export a a fast-import
stream to reposurgeon's importer. This importer does not attempt to
make changesets out of cliques of per-file commits; you'll have to do
this yourself using reposurgeon's coalesce command.

Fossil: reposurgeon declares an importer-exporter pair, and will read
from a Fossil checkout directory. It uses the native Fossil exporter,
which is pretty good but exports only versioned ignore patterns (in
.fossil-settings/ignore-glob) not the unversioned ignore-globs set
through the Web interface. The Fossil tag model has complexities that
an export to Git or other systems will not fully capture. Export also
leaves behind other data such as the wiki content, events, and
tickets.  Also be aware that Fossil does not have separate committer
and author fields - that distinction will be lost if you export to it.

darcs: reposurgeon declares an importer-exporter pair, but the
capability has only been lightly tested. There are almost certainly
undiscovered data-model issues here.  There has been no motivation to
diagnose these problems, as darcs has seen little use since 2010 and
may be extinct in the wild. The support in reposurgeon is maintained
in case it is necessary to rescue a legacy darcs history.

=== Third-tier systems

These have no coverage in the test suite.  Significant issues can be
expected.

BitKeeper (bk): As of version 7.3 (and probably earlier versions under
open-source licensing) BitKeeper has fast-import and fast-export
subcommands, and reposurgeon now knows how to use these. Note that the
BitKeeper support is untested - the exporter-importer pair may have
unknown bugs.

At one point reposurgeon was used to convert a BitKeeper history (that
of the NTP reference implementation), but that was via a custom
third-party exporter that no longer works rather than the one that was
later built in. The quality of the built-in exporter is unknown.

Monotone (mtn): reposurgeon declares an importer-exporter pair, but the
capability has not been tested. Monotone never achieved wide usage and
in 2023 appears to have been moribund for over a decade; support for
it may be dropped in a future release.

=== Indirect support

Perforce (p4): repotool can be used to mirror a remote p4 repository as a
local Git repository, and to incrementally resync the mirror; consult
the repotool manual page.  This support is experimental; it is unknown
to the author what (if any) reposurgeon cleanup operations might be
required, but a skim of Perforce documentation suggests that mapping
Perforce user IDs to a Git-style name/address pair will be desirable.

AccuRev: There are a couple of tools for translating AccuRev
repositories to live Git repositories. Of these
https://stackoverflow.com/questions/10983442/how-to-export-accurev-to-another-vcs[the
most developed appears to be called "ac2git"]; you should be able
to find it with a search engine.  We recommend you use this tool to
get a first-pass conversion to Git, then use reposurgeon to clean up
the result. The ac2git converter's goal is to produce an accurate
representation of a collection of AccuRev streams, as multiple
disconnected git branches; while there is functionality to identify
branch and merge points, actually weaving the streams into a single
DAG is something best done in reposurgeon.

=== Unsupported VCSes

https://pijul.org/[Pijul], which is still new and in intensive
development as of late 2023, will be supported when it grows an
importer/exporter pair for Git fast-import streams.

Arch, ArX, DCVS: these systems were open-source but have been dead
since 2006 (with Arch officially deprecated by its maintainer in favor
of Bazaar) and no revival of any of them seems likely. There is no
support for these and no plan to write any.

QVCS is open source but (as of late 2023) has been moribund since
2010. There is no support and no plan to write any.

Code Coop: open-source but undocumented. As of late 2023 it has been
untouched since 2018.  There is no support and no plan to write any.

AutoDesk Vault, Azure DevOps Server, CADES, Dimensions CM, ClearCase,
PVCS, StarTeam, Vault, Visual SourceSafe, Unity Version Control: These
proprietary systems are mentioned for completeness.  There is no
support for these and no plan to write any. This could change under
the following circumstances: either the vendor supplies an
exporter/importer pair, or the vendor documents a CLI sufficient for
an extractor class. These will probably never have first-class
support, unless somebody with a lot of money decides to pay for it.

[[conversion]]
== A Guide to Repository Conversion

One of the main uses for reposurgeon is converting repositories
between different version-control systems. Since around 2015
has almost always meant converting from something else to Git,
but reposurgeon keeps other possibilities open.

This section is a guide to up-converting your repository, and adopting
practices that will reduce process friction to a minimum.  It is meant
to provide context for the description of reposurgeon's features in
later sections.

If you are aiming at something other than a repository conversion,
you can safely skip this section.

In 90% of cases you'll be converting from CVS or Subversion, and those
are the cases we'll discuss in detail.

[[why-reposurgeon]]
=== Why convert with reposurgeon?

Reposurgeon is more difficult to use than any of dozens
of fully-automated conversion tools out there; you have to make
choices and compose a recipe. This section explains why it's
worth the bother.

In brief, it's because fully-automated converters don't work very
well.  They are very poor at dealing with the ontological mismatches
between the data models of different version-control systems. For
detailed discussion of the technical flaws in many common converters,
see Appendix A.

In particular: reposurgeon is the only conversion tool that handles
multi-branch Subversion repositories in full generality. It can even
correctly translate Subversion commits that alter multiple branches.

But even automated converters that are relatively good at bridging
data-model differences tend to produce crude, jackleg, unidiomatic
conversions that make the seam between the pre-conversion and
post-conversion parts of the repository very obvious.

A central example of this is commit references in change
comments. These references convey important information to anyone
reading the comments, and it is correspondingly important to change
them from using the reference format of the old system to one that is
intelligible in whatever your new one is.

As another example, git has a convention about the form of change comments;
they're supposed to consist of a standalone summary line, followed
optionally by a spacer blank line and running text.  Git relies on
this convention to produce log summaries that are easy to take in at a
glance.

Older version-control systems don't have this convention.  An ideal
conversion changes as many comments as possible to be in Git-like form so
that the Git summary tools see the data regularity they want.  But
this kind of editing can't be fully automated.  The best you can hope
for, if you want to do it right, is that your tool automates as much
of this fixup as it can *and* it assists a human operator in
applying fixups.

Neither reference-lifting nor patching comments for Git-friendliness
is a process that can be fully automated.  Both require human
judgment; accordingly, fully-automated converters don't even try to
do the right thing.  The result is often a history that is full of
unpleasant little speedbumps and distractions.  These induce
wasted developer effort and, correspondingly, higher defect rates.

On the other hand, a skilled operator of reposurgeon can produce a
conversion that is fully idiomatic in the target system, significantly
lowering future friction costs for developers browsing the history.

One fully automated reposurgeon feature of some significance that no
other importer supports is that it can parse ChangeLogs in histories
which use that Free Software Foundation convention, and use the
attributions in them to fill in Git author fields. This recovers
better information about the provenance of changesets corresponding to
patches committed by a project developer (who continues to be recorded
as the committer of that changeset).

[[commercial]]
=== Commercial Note

If you are an organization that pays programmers and has a
requirement to do a repository conversion, the author can be
mailto:esr@thyrsus.com[engaged] to perform or assist with
the transition.  You are likely to find this is more efficient than
paying someone in-house for the time required to learn the tools and
procedures. I (the author) have been very open about my methods here, but
nothing substitutes for experience when you need it done quickly
and right.

If you are wondering why it's worth spending any money at all for
a real history conversion, as opposed to just starting a new
repository with a snapshot of the old head revision, the answer
comes down to two words: risk management.

Suppose you do a snapshot conversion, head revision only. Then you get
a regression report with a way to reproduce the problem. What you want
to do is bisect in the new history to identify the revision where the
bug was introduced, because knowing what the breaking change was makes
a fix far easier. Bzzzt! You can’t. That history is missing in the
new system.

Yes, in theory you could run a manual bisection using bracketing
builds in new and old repositories. Until you have tried this, you
will have no comprehension of how easy it is to get that process
slightly but fatally wrong, and (actually more importantly) how
difficult it is _to be sure you haven’t gotten it wrong._ This is the
kind of friction cost that sounds minor until the first time it blows
up on you and eats man-weeks of NRE.

So congratulations, tracing the bug just got an order of magnitude
more expensive in engineer time, and your expected time to fix changed
proportionally. It typically only takes one of these incidents to
justify the up-front cost of having had the conversion done right.

If you go the snapshot-conversion route, maybe you’ll get lucky and
never need visibility further back. Or maybe you’ll have a disaster
because you increased the friction costs of debugging just enough that
you, say, miss a critical ship date. The more experienced with
in-the-trenches software development you are, the more plausible that
second scenario will sound.

A subtler issue is that by losing the old change comments you have
thrown away a great deal of hard-won knowledge about why your code is
written the way it is. Again, this may never matter – but if it does,
it’s going to bite you on the butt, hard, probably when you least
expect it.

And if you're thinking "No problem, the old repository will still be
around"...heh. Repositories that have become seldom-accessed are like
other kinds of dead storage in that they have a way of quietly
disappearing because after a few job turnovers the knowledge of why
they’re important is lost. Typically you don’t find out this has
happened until you have an unanticipated urgent need, at which point
whatever trouble you were in gets deeper.

Spending the relatively small amount it takes to have a proper full
conversion done right is a way of bounding your downside risk.
If you aren't a software engineer and had trouble following
the preceding argument, propose a snapshot conversion to the
engineer you trust the most and watch that person reaching for
a diplomatic way to tell you it's a stupid, shortsighted idea.

[[preparation]]
=== Step Zero: Preparation

Make sure the tools in the reposurgeon suite (especially reposurgeon
and repotool) are on your $PATH.

Create a scratch directory for your conversion work.

Run "<<repotool.adoc#,repotool initmake>>" in the scratch directory;
it requires that you follow the initmake verb with a project name.
This will create a Makefile designed to sequence your conversion, and
an empty lift script. Then set the variables near the top
appropriately for your project.

This Makefile will help you avoid typing a lot of fiddly commands
by hand, and ensure that later products of the conversion pipeline
are always updated when earlier ones have been modified or removed.

The most important variables to set in the Makefile are the ones that
set up local mirroring of your repository.  The _repotool_ command has
a mode that handles the details of making (and, when necessary,
updating) a local mirror.  To enable this you need to fill in either
REMOTE_URL or CVS_HOST and CVS_MODULE; read the header comment of the
conversion makefile for details.

If you're lifting a Subversion repository you can specify the
repository URL in either of two ways:  as a standard Subversion
repository URL (service prefix "svn:") or as an rsync URL pointing
at the same repository master directory (service prefix "rsync:").
Usually rsync mirroring is faster, but it depends on sshd running
at the server end and may being in complications around its security
features.  Use a "svn:" URL, which uses mirroring by svnsync, when you
can't make rsync work.

Note by the way that repotool mirror's "rsync:" URLs do *not* have the
requirement for a remote rsyncd that an rsync URL fed directly to
rsync itself does; internally, repotool turns them into a single-colon
host plus path rsync source specification.

If what you have is a Subversion checkout directory, the following
command will tell you the URL for its source repository. You may

--------
svn info --show-item repos-root-url
--------

If your first access attempt failed with some message about access
control, you may need to uncomment the RUSERNAME and RPASSWORD and
set them to a valid credential pair.

Later, you will put your custom commands in the lift script
file. Doing this helps you not lose older steps as you experiment with
newer ones, and it documents what you did.

Doing a high-quality repository conversion is not a simple job, and
the odds that you will get it perfectly right the first time are close
to zero. By packaging your lift commands in a repeatable script and
using the Makefile to sequence repetitive operations, you will reduce the
overhead of experimenting.

In the rest of the steps we describe below, when we write "make
foo" that means the step can be sequenced by the "foo" production
in the Makefile.  Replace $(PROJECT) in these instructions with
your project name.

You may find it instructive to type "make -n" to see what the
entire conversion sequence will look like.

[[authormap]]
=== Step One: The Author Map

You can skip this section if you're converting from a DVCS.

SCCS, RCS, CVS, and Subversion identify users by a Unix login name
local to the repository host; DVCSes use pairs of fullnames and email
addresses.  Before you can finish your conversion, you'll need to put
together an author map that maps the former to the latter; the
Makefile assumes this is named $(PROJECT).map.  The author map should
specify a full name and email address for each local user ID in the
repo you're converting. Each line should be in the following form:

----
foonly = Fred Foonly <foonly@foobar.com>
----

You can optionally specify a third field that is a timezone
description, either an ISO8601 offset (like "-0500") or a named
entry in the Unix timezone file (like "America/Chicago").  If you do,
this timezone will be attached to the timestamps on commits made by
this person.

Using the generic Makefile for Subversion, "make stubmap" will
generate a start on an author-map file as $(PROJECT).map.  Edit in real
names and addresses (and optionally offsets) to the right of
the equals signs.

How best to get this information will vary depending on your
situation.

* If you can get shell access to the repository host, looking at
`/etc/passwd` will give you the real name corresponding to
each username of a developer still active: usually you can simply
append @ and the repository hostname to each username to get a valid
email address.  You can do this automatically, and merge in real
names from the password file, using the 'repomapper' tool from the
reposurgeon distribution.

* If the repository is owned by a project on a forge site, you
can usually get the real name information through the Web interface;
try looking for the project membership or developer's list
information.

* If the project has a development mailing list, posting your
incomplete map with a request for completions often gives good results.

* If you can download the archives of the project's development
mailing list, grepping out all the From addresses may suggest some
obvious matches with otherwise unknown usernames. You may also
be able to get timezone offsets from the date stamps on the mail.
The repomapper tool can mine matching addresses from mailbox files
automatically, though it does not extract timezones.

If you are converting the repository for an open-source project, it is
good courtesy and good practice after the above first step to email
the contributors and invite them to supply a preferred form of their
name, a preferred email address to be used in the mapping, and a
timezone offset.  The reason for this is that some sites, like
https://www.openhub.net/[OpenHub], aggregate participation statistics
(and thus, reputation) across many projects, using developer name and
email address as a primary key.

Your authors file does not have to be final until you ship your
converted repo, so you can chase down authors' preferred
identifications in parallel with the rest of the work.

=== Step Two: Conversion

Install whatever front end reposurgeon needs to read your repository.
That will usually be
http://www.catb.org/~esr/cvs-fast-export[cvs-fast-export] for CVS, or
the VCS tool itself for Subversion and other systems. If you don't
have a tool you need, reposurgeon will bail out gracefully and inform
you what's missing.

The generic-workflow Makefile will call reposurgeon for you,
interpreting your $(PROJECT).lift file, when you type "make".  You may
have to watch the baton spin for a few minutes.  For very large
repositories it could be more than a few minutes.

This will convert your repository to git. If you need to export
to something else, reposurgeon has write support for several
other modern VCSes.

If you are exporting from CVS, it may be a good idea to run some trial
conversions with cvsconvert, a wrapper script shipped with
cvs-fast-export.  This script runs a conversion direct to git; the
advantage is that it can do a comparison of the repository histories
and identify problems for you to fix in your lift script.  You
probably don't want to use this for final conversion, though, as it
does not clean up CVS junk tags, perform reference lifting, or
Gitify comments.

For more detailed discussion of CVS conversion and troubleshooting see
<<CVS>>.

On the other hand, while Subversion conversion has its complexities it
is robust and well-tested. Normally reposurgeon will do a complete
branch analysis for you.  On most Subversion repositories, and in
particular anything with a standard trunk/tags/branches layout, it
will do the right thing. It will also cope with adventitious branches
in the root directory of the repo, such as some projects use for
website content.

There is, however, a minor problem around tags, and a slightly
more significant problem around Subversion merges. Also, some
Subversion repositories are multi-project with a nonstandard
directory layout. 

For more detailed discussion of Subversion conversion and
troubleshooting see <<subversion>>.  For discussion of handling
multiproject repositories, see <<multiproject>>.

=== Step Three: Sanity Checking

Before putting in further effort on polishing your conversion
and putting it into production, you should check it for basic
correctness.

Pay attention to error messages emitted during the lift.  Most
of these, and remedial actions to take, are described in this
guide.

For Subversion lifts, use the "headcompare", "tagscompare" and
"branchescompare" productions to compare the converted with the
unconverted repository.  If you didn't use the cvsconvert wrapper for
your CVS lift, these productions have a similar effect.  Be aware
that these operations may be extremely slow on large Subversion
repositories.

The only differences you should see are those due to keyword expansion
and ignore-file lifting.  If this is not true, you may have found a
serious bug in either reposurgeon or the front end it used, or you
might just have a misassigned tag that can be manually fixed. Consult
<<reporting-bugs>> for information on how to usefully report bugs.

Use reposurgeon's '```<<lint_cmd>>```' command to find anomalies like
detached branches that may need manual correction.

If you are converting from CVS, use reposurgeon's
'```<<view_cmd>>```' command to examine the conversion, looking (in
particular) for misplaced tags or missing branch joins. Often these can be
manually repaired with little effort. These flaws do 'not' necessarily
imply bugs in cvs-fast-export or reposurgeon; they may simply indicate
previously undetected malformations in the history. However, reporting
them may help improve cvs-fast-export.

[[cleanup]]
=== Step Four: Cleanup

You should now have a git repository, but it is likely to have a
lot of cruft and conversion artifacts in it.

Here's a checklist of cleanup steps, which we'll expand on later in
this section. If you're using the makefile generated by repotool, some
of these will be done by commands in your lift script.

==== Apply your author map

This step applies to SCCS, RCS, CVS, and SVN. Map author IDs from
local to DVCS form.  The reposurgeon command for this is "authors
read"; see '```<<authors_cmd>> read```'.

==== Mine changelogs

If you don't know what a GNU ChangeLog is you can skip this step.

If you're converting form a VCS that doesn't have separate committer
and author attributions (e.g. CVS, SVN, Fossil), and the project
you're converting used the GNU ChangeLog convention, include
"<<changelogs_cmd>>" somewhere in your lift script to capture
authorship information.

The command:

--------
changelogs
--------

==== Clean up after legacy character sets

You may have metadata in your repository in an encoding incompatible
with UTF-8. The most common glitch of this kind is Latin-1 characters
in committer names.

Unfortunately, this process can't be automated; the fact that the
encoding of a text in a legacy character set like Latin-1 or GB18030
can't be deduced just by looking at the bits is exactly what led
to the development of Unicode and UTF-8.

The selector `=I` will choose all commits with attributions and
comments that don't decode to UTF-8 in both the commit comment and
attribution parts. You can use `=I list inspect` to view them.

It's up to you to figure out what the encoding is and apply the
'```<<transcode_cmd>>```' command to re-encode in UTF-8. You can test
the results of applying different encodings using the --decode
option of the '```<<list_cmd>>```' or '```<<msgout_cmd>>```'
commands.

Example commands:

--------
# Try out Shift_JIS decoding to see if it throws errors
=I msgout --decode=Shift_JIS

# Apply latin1 decoding to UTF-8
=I transcode latin1
--------

==== Convert ignore files

If you're using reposurgeon for a CVS- or Subversion-to-git conversion
cvs-fast-export and reposurgeon will convert ignore patterns for you
automatically, both expressing Subversion svn:ignore and
svn:global-ignores properties as .gitignore files and lifting
.cvsignore files to .gitignore files.

Any .gitignore files found in a Subversion repository *before
conversion* were almost certainly created by git-svn users ad-hoc and
will be discarded by the importer; it is up to the human doing the
conversion to look through them and rescue any ignore patterns that
should be merged into the converted repository.  This behavior can be
reversed with the `--user-ignores` option, which will retain that
information and merge it with the ignore patterns generated from
svn:ignore and svn:global-ignores properties.

Other conversions may need this command:

--------
ignores --translate --defaults
--------

==== Remove junk tags

Your conversion may contain mechanically-generated tags (both
lightweight and annotated) that convey no actual information.  There
are clutter and should be removed

You can examine the repository's tags with

--------
list tags
--------

CVS makes tags that are branch root markers for internal
purposes. These are not deleted automatically by cvs-fast-export
because implicit magic that deletes data generally turns out to be a
bad idea. They often persist through up-conversions to SVN. This
command will clean these up:

--------
delete tag /-root$/
--------

SVN commits with no fileops are automatically transformed into
annotated tags when reading a Subversion repository. These are
especially likely to occur if you are converting a Subversion
repository that had a shady past as a CVS tree and was converted using
cvs2svn. This command will clean these up:

--------
delete tag /emptycommit/
--------

==== Remove junk commits

Importers may generate commits that are empty (have no fileops) for
various reasons. You will probably want to delete these; they're
preserved just in case something about the metadata is interesting.

You can review empty commits with:

--------
=Z list
--------

You can save any that have interesting comments as tags attached
to their parent commit with '```<<tagify_cmd>>```'.

The rest of this instruction step applies only if you are
converting a Subversion repository.

Junk generated by *cvs2svn* to carry tag information lurk in the
history of many Subversion projects. When these junk commits
are empty, either the importer will already have tagified them
(in which case you cleaned them up in the previous step) or
you just caught them with "=Z list".

Less commonly, generated junk commits have long lists of spurious
delete fileops.  These can be listed with the =D selector:

--------
=D list
--------

The trickiest kind has actual file content duplicating parent file
versions, or referring randomly to file versions far older than the
junk commit.  You can sometimes spot these because they have commit
comments that are empty or consist of the string "*** empty log
message ***",

Either <<view_cmd>> or <<graph_cmd>> can be q good way to spot junk commits.
Use them to eyeball the picture of the commit DAG created by the reposurgeon
'graph' command - they tend to stand out visually as leaf nodes in odd
places. Be aware that the <<graph_cmd>> command outputs DOT, the
language interpreted by the http://www.graphviz.org/[graphviz] suite;
you will need a DOT rendering program and an image viewer.
Unfortunately, for large repositories the sheer size of the graph image
makes this impractical.

==== Fix up references in commit comments

Different version-control systems have different conventions for how
a commit comment can refer to a previous commit. Modern DVCSes
usually use opaque hexadecimal strings that express some hash
function of the content of the commit and its ancestors. Subversion,
which is centralized, uses integer revision numbers starting from 1.
CVS occasionally uses per-file versions that look like numerals
separated by dots.

What all these have in common is that they stop being useful when you
move your history to a new version-control system.  There are
two different ways to deal with this, and you need to make a
policy decision about which fits your needs.

One way is to use '```<<append_cmd>>```' to add a line to each commit
giving its identifier. This allows you to chase references by
searching in the text of comments. This is a good choice if you have
references to commits outside the repository (say, in mailing-list
archives) that you want to stay valid.

The other way is to convert these references into action stamps -
identifiers that are independent of the VCS you are in,  This has the
advantage of being future-proof and cluttering your conversion less. 

You due the latter thing with '```<<stampify_cmd>>```',  But there
is a preparation step you have to do first, which is to hack the
existing references into reference cookies that
'```<<stampify_cmd>>```' can see.

There's a detailed of this process and its commands at
<<reference-lifting>>.

==== Massage comments into summary-line-plus-continuation form.

git and hg both encourage comments to begin with a summary line that
can stand alone as a short description of the change; this practice
produces more readable output from `git log` and `hg log`.

For a really high-quality conversion, multi-line comments should be
edited into this form. Some of this can be done with,
'```<<gitify_cmd>>```', but you'll need to go through and fix
remaining cases by hand.  The =L selector is good for finding these.

Here is a workflow for this cleanup step:

1. Do "=L msgout --id >message.box" to dump comments that lack
a spacer line after the first to a mailbox file.  The actual name
of the mailbox file doesn't matter of course.

2. Use your preferred text editor to fix up the comments.

3. Do "msgin <message.box" to apply the fixes.

==== Patch in missing branch merges.

You can skip this step if you're converting from a DVCS, it will
already have branch merges expressed.  But CVS has no concept of
merging branches at all. Under Subversion, reposurgeon tries to
interpret svn:mergeinfo properties to create merges automatically, but
Subversion merging is tricky enough that these properties are often
not set when they could have been.

You can review your branch tip commits with the =H selector. By
examining commits near in time to each tip, you may be able to
identify when there is an unexpressed branch merge.  Under Subversion,
look for svn:mergeinfo properties as clues (but see the warnings
about this at <<subversion>>).

You can use '```<<merge_cmd>>```' to fix these up. This is safe
because adding a merge link doesn't modify any later content;
it's only used for ancestry tracking when computing diffs.

====  Subversion only: Review branch tip deletes and deletealls

In Subversion it is common practice to delete a branch directory when
that line of development is finished or merged to trunk; this makes
sense because it reduces the checkout size of the repo in later
revisions.

In a DVCS, deletes at a branch tip don't save you any storage, so it
makes more sense to leave the branch with all of its tip content live
if you're not going to delete it entirely.  

It;s a judgment call whether to delete these. They could be considered
clutter, or they could be considered documentation of the fact that
development on a branch has closed.

==== CVS only: Resolve zombie files

CVS conversions occasionally have <<zombie>> files - that is, files
which persist in conversion commits after they were
deleted in the CVS repository.

The easiest way to detect these is to run a trial conversion with
cvsconvert(1).  It will report content mismatches, including this
kind. You will be able to fix them by patching D fileops into the
history.

There is a more detailed discussion of this issue at <<CVS>>.

==== CVS only: remove ~/.cvsrc history (optional)

Optionally, do this to remove a CVS relic

--------
delete path .cvsrc
--------

This file, if it existed, set options for CVS commands. It will
not be relevant in your new VCS and is just clutter.

==== Subversion only: check for a root branch

After conversion of a branchy Subversion repository, look to see if
there is a 'root' branch.  If there are any commits with a
sufficiently odd that reposurgeon can't figure out what branch they
belong to, they'll wind up there.

This command will tell you if you have a root branch in your conversion:

--------
list names
--------

If this happens, it's likely those commits will be junk.  It's up to
you to delete or transplant them appropriately.

==== Fix up or remove $-keyword cookies in the latest revision.

One minor feature you lose in moving from SCCS, RCS, CVS, Subversion,
or BitKeeper to a DVCS is keyword expansion. There was a practice of
embedding magic cookies in master files that would be expanded on
checkout with various metadata like the commit date and committer
ID. These are useless clutter under modern VCSes and should be
removed.

You should go through the last revision of the code and remove `$Id$`,
`$Date$`, `$Revision$`, and other keyword cookies lest they become
unhelpful relics. The full Subversion set is `$Date$`, `$Revision$`,
`$Author$`, `$HeadURL$` and `$Id$`. CVS uses `$Author$`, `$Date$`,
`$Header$`, `$Id$`, `$Log$`, `$Revision$`, also (rarely) `$Locker$`,
`$Name$`, `$RCSfile$`, `$Source$`, and `$State$`. A command like `grep
-R '$[A-Z]' .` may be helpful.

==== Run lint to detect remaining anomalies

Run <<lint_cmd>> to detect remaining anomalies that might need to be
patched.

==== Record the conversion into the history

It's good practice to leave an annotated tag at the conversion point
noting the date and time of the repo lift.  See the next section on
conversion comments for discussion.

Here's an example of how to make an informative conversion tag:

--------
msgin --create <<EOF
Tag-Name: git-conversion

Marks the spot at which this repository was converted from Subversion to git.

Conversion notes are enclosed in double square brackets. Junk commits
generated by cvs2svn have been removed, commit references have been
mapped into a uniform VCS-independent syntax, and some comments edited
into summary-plus-continuation form.
EOF
--------

Experiments with reposurgeon suggest that `git import` doesn't try to
pack or otherwise optimize for space when it populates a repo from a
dump file; this produces large repositories.  Running `git repack` and
`git gc --aggressive` can slim them down quite a lot.

==== Garbage-collect the conversion

If your target was git, run  `git gc --aggressive`. This can reduce it
in size considerably.

[[conversion-comments]]
==== A note on conversion comments

Sometimes, in converting a repository, you may need to insert an
explanatory comment - for example, if metadata has been garbled or
missing and you need to point to that fact.

It's helpful for repository-browsing tools if there is a uniform
syntax for this that is highly unlikely to show up in repository
comments.  Enclosing translation notes in [[ ]] is recommended; this
has the advantage of being visually similar to the [ ] traditionally
used for editorial comments in text.

[[recovering]]
==== A note on recovering from errors

Occasionally you'll discover problems with a conversion after you've
pushed it to a project's hosting site, typically to a bare repo that
the hosting software created for you.  Here's how to cope:

1. Do your surgery on a copy of the repo with its .git/config
pointing to the public location.

2. Warn the public repo's users that it is briefly going out
of service, and they will need to re-clone it afterwards!

3. Ensure that it is possible to force-push to the repository.
How you do this will vary depending on your hosting site.

4. On gitlab.com, under Settings, there is a "Protected Branches"
item you can use. If you unprotect a branch, you can force-push to it.
+
Elsewhere, you may be able to re-initialize the public repo (this
works, for example, on SourceForge). You'll need ssh access to the
bare repo directory on the host - let's suppose it's 'myproject'. Pop
up to the enclosing directory and do this:
+
----
    mv myproject myproject-hidden
    rm -fr myproject-hidden/*
    git init --bare myproject-hidden
    mv myproject-hidden myproject
----
+
The point of doing it this way is (a) so you never actually remove
myproject (on many hosts you will not have create permissions in the
enclosing directory), and (b) so no user can update the repo while
you're clearing it (mv is atomic).
+
Here's a script that will do the job on SourceForge:
+
----
#!/usr/bin/expect -f
#
# nuke - nuke a SourceForge repo
#
# usage: nuke project [userid]
#

if {$argc < 1} {
    puts "nuke: project name argument is required"
    exit 1
} else {
    set project [lindex $argv 0]
    set user $env(USER)
    if {$argc >= 2} {
	set user [lindex $argv 1]
    }
}

set remoteprompt "bash-4.1"

set timeout -1
spawn $env(SHELL)
match_max 100000
send -- "ssh -t $user@shell.sourceforge.net create"
expect -exact "ssh -t $user@shell.sourceforge.net \r create"
send -- "\r"
expect -exact "$remoteprompt\$ "
send -- "cd /home/git/p/$project\n"
expect -exact "$remoteprompt\$ "
send -- "cd git-main.git\n"
expect -exact "$remoteprompt\$ "
send -- "rm -fr *\n"
expect -exact "$remoteprompt\$ "
send -- "git init --bare .\n"
expect -exact "$remoteprompt\$ "
----
+
After re-initializing, you should be able to run `git
push` to push the new history up to the public repo.

5. From your modified local repo, try
+
----
     git push --mirror --force
----
+
to push the new history up to the public repo.

6. Inform the public repo's users that it is available and remind
them that they will need to re-clone it.

On GitLab, you can get a similar effect by unprotecting all branches
and doing a git push --force to unconditionally overwrite the
public history.  It is good practice to re-protect the branches
afterwards.

=== Step Five: Client Tools

Developers who are already git fans and know how to use a git
client will, of course, have no particular trouble using a git
repository.

Windows users accustomed to working through TortoiseSVN can move to
https://tortoisegit.org/[TortoiseGIT].

If you have developers attached to the CVS interface, it is possible
(and in fact relatively easy) to set up a gateway interface that lets
them continue using their CVS client tools.  Consult the documentation
for
https://www.kernel.org/pub/software/scm/git/docs/git-cvsserver.html[git-cvsserver].

=== Step Six: Good Practice

Educate your developers in the following good practices:

==== Commit references

The combination of a committer email address with a ISO8601 timestamp
is a good way to refer to a commit without being VCS-specific.  Thus,
instead of "commit 304a53c2", use
"<2011-10-25T15:11:09Z!fred@foonly.com>". It is recommended that you
_not vary from this format_, even in trivial ways like omitting the
'Z' or changing the 'T' or '!'. Making these cookies uniform and
machine-parseable will have good consequences for future
repository-browsing tools. The reference-lifting code in reposurgeon
generates them.

Being careful about this has an additional benefit.  Someday your
project may need to change VCSes yet again; on that day, it will be
extremely helpful if nobody has to try to convert years' or decades'
worth of VCS-specific magic cookies in the history.

Sometimes it's enough to quote the summary line of a commit. So,
instead of "Back out most of commit 304a53c2", you might write
"Back out Attempted divide-by-zero fix."

When appropriate, "my last commit" is simple and effective.

==== Comment format

As previously noted, git and hg both want comments to begin with a
summary line that can stand alone as a short description of the
change; this may optionally be followed by a separating blank line
and details in whatever form the commenter likes.

Try to end summary lines with a period.  Ending punctuation other
than a period should be used to indicate that the summary line is
incomplete and continues after the separator; "..." is conventional.

For best results, stay within 50 characters for summary lines and
within 72 characters for detail lines.  Don't go over 80.  The Linux
kernel developers like to stay at 65 or under.

Good comment practice produces more readable output from `git
log` and `hg log`, and makes it easy to take in
whole sequences of changes at a glance.

Consider adopting the guidelines at
https://www.conventionalcommits.org/en/v1.0.0/[Conventional Commits]

[[theory]]
== Theory of Operation

[[outside]]
=== The outside view

As the quick-start example shows, you're typically going to do three
steps when you use reposurgeon: (1) read in one (or more)
repositories, (2) do surgical things on them, and (3) write out one
(or more) repositories.

To keep reposurgeon simple and flexible, it normally does not do its
own repository reading and writing.  Instead, it relies on being able
to parse and emit the command streams created by git-fast-export and
read by git-fast-import.  This means that it can be used on any
version-control system that has both fast-export and fast-import
utilities. The git-import stream format also implicitly defines a
common language of primitive operations for reposurgeon to speak.

In order to deal with version-control systems that do not have
fast-export equivalents, reposurgeon can also host extractor code that
reads repositories directly.  For each version-control system
supported through an extractor, reposurgeon uses a small amount of
knowledge about the system's command-line tools to (in effect) replay
repository history into an input stream internally. Repositories under
systems supported through extractors can be read by reposurgeon, but
not modified by it.  In particular, reposurgeon can be used to move a
repository history from any VCS supported by an extractor to any VCS
supported by a normal importer/exporter pair.

Mercurial repository reading is implemented with an extractor
class; writing is handled with the "hg-git-fast-import" command.  A
test extractor exists for git, but is normally disabled in favor of
the regular exporter.

Subversion is an important exception.  Its exporter is '```svnadmin
dump```', which doesn't ship a git-fast-import stream, but rather the
unique dump format supported by Subversion. Reposurgeon contains
an interpreter for this stream format.

As a matter of historical interest, some old versions of reposurgeon
had the ability to build a Subversion repository on output by
synthesizing a Subversion dump stream and feeding it to '```svnadmin
load```'.  This feature was a cute stunt, but was abandoned during
translation to Go for a couple of reasons. Most importantly, there is
zero demand for moving histories *to* Subversion - and supposing there
were, moving content and metadata from git's DAG representation to a
Subversion stream is very lossy. Subversion to Git to Subversion wouldn't
even have round-tripped well.

[[inside]]
=== The inside view

Between reads and writes, reposurgeon can usefully be thought of as
a structure editor for directed acyclic graphs with a pre-defined set of
attributes on their nodes.

To get a feel for what that graph is like, it's helpful to have seen a
git-fast-import stream file.  Here is a trivial example from the
reposurgeon test suite, describing a history with two commits to a single
file:

--------
blob
mark :1
data 20
1234567890123456789

commit refs/heads/master
mark :2
committer Ralf Schlatterbeck <rsc@runtux.com> 0 +0000
data 14
First commit.
M 100644 :1 README

blob
mark :3
data 20
0123456789012345678

commit refs/heads/master
mark :4
committer Ralf Schlatterbeck <rsc@runtux.com> 10 +0000
data 15
Second commit.
from :2
M 100644 :3 README

--------

A git-fast-import stream consists of a sequence of commands
which must be executed in the specified sequence to build the repo; to
avoid confusion with reposurgeon commands
we will refer to the stream commands as _events_ in
this documentation.  These events are implicitly numbered from 1
upwards.  Most commands require specifying a selection of event
sequence numbers so reposurgeon will know
which events to modify or delete.

For all the details of event types and semantics, see the
git-fast-import(1) manual page; the rest of this paragraph is a quick
start for the impatient. The most prominent events in a stream are
*commits* describing revision states of the repository; these group
together under a single change comment one or more *fileops* (file
operations), which usually point to *blobs* that are revision states
of individual files. A fileop may also be a delete operation
indicating that a specified previously-existing file was deleted as
part of the commit; there are a couple of other special fileop types of
lesser importance.

Reposurgeon's internal representation of a repository history is
basically a deserialized git fast-import stream.  A few extra
attributes are supported; most notably, commits and resets have a
legacy-id attribute that carries over the object's ID from whatever
version-control system exported the stream, in particular a Subversion
or CVS revision number.

[[interpreter]]
=== The interpreter view

The program can be run in one of two modes, either as an
interactive command interpreter or in batch mode to execute commands
given as arguments on the reposurgeon invocation line.

The only differences between these modes are (1) the interactive one
begins by turning on the '```interactive```' option, (2) in batch mode
all errors (including normally-recoverable errors in selection-set
syntax) are fatal, and (3) each command-line argument beginning with
'```--```' has that stripped off (which in particular means that
`--help` and `--version` will work as expected).

Also, in interactive mode, Ctrl-P and Ctrl-N will be available to
scroll through your command history, and tab completion of command
keywords, options, and arguments (wherever that makes semantic sense) is
available. Entering a tab just after the command keyword will show you
subcommands and options. 

Commands that modify the history don't normally generate summaries
of what bits they have changed. Instead there is a general mechanism
called "Q bits" that helps you find and examine recently-modified
things. See <<qbits>> for details.

It is expected that interactive mode will be used mainly for exploring
repository metadata, while conversion experiments will be captured in
a script that is gradually improved until the day final cutover
can be performed and the old repository decommissioned.

Note that this means the old repository can be left in service while
the conversion recipe is under development. Recipe development should
be treated as a serious project with its own change tracking.

// START-TOC

== The Command Interpreter

[[command-syntax]]
=== Command syntax

To give you the flavor of the command language, here
are some simple examples:

--------
29..71 list             ;; list summary index of events 29..71.

236..$ list             ;; List events from 236 to the last.

<#523> list inspect     ;; Look for commit #523; they are numbered
                        ;; 1-origin from the beginning of the
                        ;; repository.

<2317> list inspect     ;; Look for a tag with the name 2317, a tip
                        ;; commit of a branch named 2317, or a commit
                        ;; with legacy ID 2317. Inspect what is found.
                        ;; A plain number is probably a legacy ID
                        ;; inherited from a Subversion revision
                        ;; number.

/regression/ list       ;; list all commits and tags with comments or
                        ;; committer headers or author headers
                        ;; containing the string "regression".

1..:97 & =T delete      ;; delete tags from event 1 to mark 97.

[Makefile] list inspect ;; Inspect all commits with a file op touching
                        ;; Makefile and all blobs referred to in a
                        ;; fileop touching Makefile.

@dsc(:55) list          ;; Display all commits with ancestry tracing
                        ;; to :55.

@min([.gitignore]&=C) remove .gitignore
                        ;; Remove the first .gitignore fileop in the
                        ;; repo.  In a Subversion lift this contains
                        ;; patterns corresponding to Subversion default
                        ;; ignores.
--------

// TOPIC
include::docinclude/syntax.adoc[]

When a command changes repository state, it will usually so indicate
in a response.

[[wayfinding]]
=== Finding your way around

Help is always available.

// COMMAND
include::docinclude/help.adoc[]

Command history is always available.

// COMMAND
include::docinclude/history.adoc[]

You don't need to exit the interpreter to run quick shell commands.

// COMMAND
include::docinclude/shell.adoc[]

Here's how to bail out:

// COMMAND
include::docinclude/quit.adoc[]

[[regular_expressions]]
=== Regular expressions

Regular expressions are an important building block of
the command language, used both in event selections and
various command arguments.

// TOPIC
[[regexp]]
include::docinclude/regexp.adoc[]

[[selections]]
=== Selection syntax

// END-TOC

Commands to reposurgeon consist of a command keyword, usually preceded
by a selection set, sometimes followed by whitespace-separated
arguments. It is often possible to omit the selection-set argument and
have it default to something reasonable.  For commands that are
considered safe (no side effects) the default is all events; for risky
commands the default is no events.

A selection set is ordered; that is, any given element may occur
only once, and the set is ordered by when its members were first added.

The selection-set specification syntax is an expression-oriented
mini-language.  The most basic term in this language is a
location. The following sorts of primitive locations
are supported:

event numbers::
   A plain numeric literal is interpreted as a 1-origin
   event-sequence number. It is not expected that you will
   have to use this feature often.

marks::
   A numeric literal preceded by a colon is interpreted
   as a mark; see the import stream format documentation for explanation
   of the semantics of marks.

tag and branch names::
   The basename of a branch (including branches in the refs/tags
   namespace) refers to its tip commit.  The name of a tag is equivalent
   to its mark (that of the tag itself, not the commit it refers to). Tag
   and branch locations are bracketed with `<` `>` (angle brackets) to
   distinguish them from command keywords.

legacy IDs::
   If the content of name brackets (`< >`) does not match a tag or
   branch name, the interpreter next searches legacy IDs of commits.
   This is especially useful when you have imported a Subversion dump; it
   means that commits made from it can be referred to by their
   corresponding Subversion revision numbers.

commit numbers::
   A numeric literal within name brackets (`< >`)
   preceded by `#` is interpreted as a 1-origin
   commit-sequence number.

reset targets::
   If the previous ways of interpreting a name within brackets don't
   resolve, the name is checked to see if it matches a reset. If so,
   the expression resolves to the commit the reset is attached to.

reset@ names::
   A name with the prefix '```reset@```' refers to the latest reset with a
   basename matching the part after the `@`. Usually there is only one such
   reset.

$::
   Refers to the last event.

These may be grouped into sets in the following ways:

ranges::
   A range is two locations separated by '```..```', and is
   the set of events beginning at the left-hand location and ending at the
   right-hand location (inclusive).

lists::
   Comma-separated lists of locations and ranges are
   accepted, with the obvious meaning.

There are some other ways to construct event sets:

visibility sets::
   A visibility set is an expression specifying a set of event
   types. It will consist of a leading equal sign, followed by type
   letters. These are the type letters:
+
[options=""]
|===================================================================
| B | blobs                            | Most default selection sets exclude
blobs; they have to be manipulated through the commits they are attached to.
| C | commits                          |
| D | all-delete commits               | These are artifacts produced by some
older repository-conversion tools.
| E | earliest commit on each branch   |
| F | all fork (multi-child) commits   |
| H | head (branch tip) commits        |
| I | all commits not decodable to UTF-8 |
| J | all commits with non-ASCII (possible ISO 8859) characters |
| L | commits with unclean multi-line comments |  E.g. without a separating
empty line after the first
| M | merge (multi-parent) commits     |
| N | Legacy IDs                       | Any comment matching a cookie
(legacy-ID) format
| O | orphaned (parentless) commits    |
| P | passthroughs                     | All event types simply passed through,
including comments, `progress` commands,
and `checkpoint` commands
| Q | Recently touched                 | Set/cleared by many commands
| R | all resets                       |
| T | all tags                         |
| U | commits with callout parents     |
| X | commits in a bisection triple    } Earliest/middle/latest
| Z | commits with no fileops          |
|===================================================================

references::
   A reference name (bracketed by angle brackets) resolves
   to a single object, either a commit or tag.
+
[options="header"]
|===================================================================
| type          | interpretation
| tag name      | annotated tag with that name
| branch name   | the branch tip commit
| legacy ID     | commit with that legacy ID
| assigned name | name equated to a selection by <<assign_cmd>>
|===================================================================
+
Note that if an annotated tag and a branch have the same name foo,
`<foo>` will resolve to the tag rather than the branch tip commit.

dates and action stamps::
   A date or action stamp in angle brackets resolves
   to a selection set of all matching commits.
+
[options="header"]
|===================================================================
| type                           | interpretation
| RFC3339 timestamp              | commits or tags with that time/date
| action stamp <timestamp!email> | commits or tags with that timestamp and
author (or committer if no author). Aliases of the author are also accepted.
| yyyy-mm-dd part of RFC3339 timestamp | all commits and tags with that date
|===================================================================
+
To refine the match to a single commit, use a 1-origin index
suffix separated by `#`. Thus `<2000-02-06T09:35:10Z>` can
match multiple commits, but `<2000-02-06T09:35:10Z#2>` matches
only the second in the set (by position in the stream file).

text search::
    A text search expression is a regular expression
    surrounded by forward slashes (to embed a forward slash in it, use a
    C-like string escape such as `\x2f`).
+
A text search normally matches against the comment fields of commits and
annotated tags, or against their author/committer names, or against
the names of tags; also the text of passthrough objects.
+
The scope of a text search can be changed with qualifier
letters after the trailing slash.  These are as follows:
+
[options="header"]
|===================================================================
| letter   | interpretation
| a        | author name in commit
| b        | branch name in commit; also matches blobs referenced by commits
on matching branches, and tags which point to commits on matching branches.
| c        | comment text of commit or tag
| r        | committish reference in tag or reset
| p        | text in passthrough
| t        | tagger in tag
| n        | name of tag
| B        | blob content
|===================================================================
+
Multiple qualifier letters can add more search scopes.
+
(The "b" qualifier replaces the branch-set syntax
in earlier versions of reposurgeon.)

paths::
   A "path expression" enclosed in square brackets resolves to the
   set of all commits and blobs related to a path matching the given
   expression. The path expression itself is either a path literal or a
   regular expression surrounded by slashes. Immediately after the
   trailing / of a path regexp you can put any number of the following
   characters which act as flags: '```a```', '```c```', '```D```', '```M```',
   '```R```', '```C```', '```N```'.
+
If the first character in a path expression is '```~```', the path
expression is negated; that is, it is evaluated for "not matching"
rather than "matching".
+
By default, a path is related to a commit if the latter has a
fileop that touches that file path - modifies that change it, deletes
that remove it, renames and copies that have it as a source or
target. When the '```c```' flag is in use the meaning changes: the paths
related to a commit become all paths that would be present in a
checkout for that commit.
+
A path literal matches a commit if and only if the path literal is
exactly one of the paths related to the commit (no prefix or suffix operation
is done). In particular a path literal won't match if it corresponds to a
directory in the chosen repository.
+
A regular expression matches a commit if it matches any path related to
the commit anywhere in the path. You can use `^` or `$` if you want the
expression to only match at the beginning or end of paths.  When the '```a```'
flag is in use, the path expression selects commits whose every path matches the
regular expression. This is necessarily a subset of commits selected without the
'```a```'
flag because it also selects commits with no related paths (e.g. empty commits,
deletealls and commits with empty trees). If you want to avoid those, you can
use e.g. '```[/regexp/] & [/regexp/a]```'.
+
The flags '```D```', '```M```', '```R```', '```C```', '```N```' restrict match
checking to the corresponding fileop types.  Note that this means an '```a```'
match is
easier (not harder) to achieve.  These are no-ops when used with '```c```'.
+
A path or literal matches a blob if it matches any path that
appeared in a modification fileop that referred to that blob. To
select purely matching blobs or matching commits, compose a path
expression with `=B` or `=C`.
+
If you need to embed '```[^/]```' into your regular expression (e.g. to
express "all characters but a slash") you can use a C-like string
escape such as `\x2f`.

// START-TOC

// TOPIC
include::docinclude/selection.adoc[]

// TOPIC
include::docinclude/functions.adoc[]

// TOPIC
include::docinclude/operators.adoc[]

[[redirection]]
=== Redirection and shell-like features

// TOPIC
include::docinclude/redirection.adoc[]

[[qbits]]
=== Q bits

Each event has an associated "Q" bit that says whether it was touched
by the last Q-aware command.  Almost all commands that modify the
history set Q bits in some interesting way; so do some report
generators, such as the "lint" command.

Thus, "=Q count" and "=Q list" can provide you with a check on the
scope of modifications while you are trying things out.  Or you can
use "=Q list" and "=Q list inspect" to zero in on anomalies reported by
lint.

[[import-export]]
== Import and Export

reposurgeon can hold multiple repository states in memory.  Each has a
name.  At any given time, one may be selected for editing. Commands in
this group import repositories, export them, and manipulate the
in-core list and the selection.

If you are planning a conversion from Subversion, you should
probably read <<subversion>> after this section.

If you are planning a conversion from Mercurial, out should
probably read <<mercurial>> after this section.

=== Reading and writing repositories

[[read_cmd,read]]
// COMMAND
include::docinclude/read.adoc[]

// COMMAND
include::docinclude/write.adoc[]

[[preferences]]
=== Repository type preference

// COMMAND
include::docinclude/prefer.adoc[]

// COMMAND
include::docinclude/select.adoc[]

[[rebuild]]
=== Rebuilds in place

reposurgeon can rebuild an altered repository in place.  Untracked
files are normally saved and restored when the contents of the new
repository are checked out (but see the documentation of the
'```<<preserve_cmd>>```' command for a caveat).

// COMMAND
include::docinclude/rebuild.adoc[]

[[recovery]]
=== Crash recovery

This section will become relevant only if `reposurgeon` or something
underneath it in the software and hardware stack crashes while in the
middle of writing out a repository, in particular if the target
directory of the rebuild is your current directory.

The tool has two conflicting objectives. On the one hand, we
never want to risk clobbering a pre-existing repo.  On the other hand,
we want to be able to run this tool in a directory with a repo and
modify it in place.

We resolve this dilemma by playing a game of three-directory monte.

. First, we build the repo in a freshly-created staging
directory.  If your target directory is named
`/path/to/foo`, the staging directory will be a
peer named `/path/to/foo-stageNNNN`, where NNNN is
a cookie derived from reposurgeon's process
ID.

. We then make an empty backup directory.  This directory will
be named `+/path/to/foo.~N~+`, where _N_ is incremented
so as not to conflict with any existing backup directories.
reposurgeon never, under any circumstances,
ever deletes a backup directory.
+
So far, all operations are safe; the worst that can happen up to
this point if the process gets interrupted is that the staging and
backup directories get left behind.

. The critical region begins. We first move everything in the
target directory to the backup directory.

. Then we move everything in the staging directory to the
target.

.  We finish off by restoring untracked files in the target
directory from the backup directory. That ends the critical
region.

During the critical region, all signals that can be ignored are
ignored.

[[preservation]]
=== File preservation

When the repository type you are working with has a "lister" method,
it can tell which files in a repository directory are not checked in
and will copy them into the edited repository made by a rebuild.

The following commands are required only if there is no lister method
and you have to set preservations by hand. Under systems with such a
command (which include git and hg), all files that are neither beneath
the repository dot directory nor under reposurgeon temporary
directories are preserved automatically.

// COMMAND
include::docinclude/preserve.adoc[]

// COMMAND
include::docinclude/unpreserve.adoc[]

[[tarballs]]
=== Incorporating data

When converting a legacy repository, it sometimes happens that
there are archived releases of the project surviving from before the
date of the repository's  initial commit.  It may be desirable to
insert those releases at the front of the repository history.
Do do this, use this command:

// COMMAND
include::docinclude/import.adoc[]

[[lists]]
=== The repository list

Reposurgeon can have several repositories loaded at once.
The following commands operate on the repository list.

// COMMAND
include::docinclude/choose.adoc[]

// COMMAND
include::docinclude/drop.adoc[]

// COMMAND
include::docinclude/clone.adoc[]

The "rename repo" mode of <<help_cmd>> can be used to rename
a repository.

[[information]]
== Information and reports

Commands in this group report information about the selected
repository.

The output of these commands can individually be redirected to
a named output file. Where indicated in the syntax, you can prefix the
output filename with '```>```' and give it as a following
argument.  If you use '```>>```' the file is opened
for append rather than write.

[[reports]]
=== Reports on the DAG

// COMMAND
include::docinclude/list.adoc[]

// COMMAND
include::docinclude/graph.adoc[]

// COMMAND
include::docinclude/view.adoc[]

// COMMAND
include::docinclude/lint.adoc[]

[[statistics]]

// COMMAND
include::docinclude/count.adoc[]

[[examining-tree-states]]
=== Examining tree states

// COMMAND
include::docinclude/checkout.adoc[]

// COMMAND
include::docinclude/diff.adoc[]

[[surgical]]
== Surgical Operations

These are the operations the rest of reposurgeon is designed
to support.

=== Creations, deletions, and renames

// COMMAND
include::docinclude/create.adoc[]

[[squash_cmd,squash]]
[SELECTION] squash [POLICY...]::
   Combine or delete commits in a selection set of events. The
   default selection set for this command is empty. Has no effect on
   events other than commits unless the `--delete` policy is selected;
   see the '```<<delete_cmd>>```' command for discussion.
+
Normally, when a commit is squashed, its file operation list
(and any associated blob references) gets either prepended to the
beginning of the operation list of each of the commit's children or
appended to the operation list of each of the commit's parents. Then
children of a deleted commit get it removed from their parent set and
its parents added to their parent set.
+
The analogous operation is performed on commit comments, so no
comment text is ever outright discarded.  Exception: comments
consisting of "```++*** empty log message ***++```", as generated by
CVS, are ignored.
+
The default is to squash forward, modifying children; but see the
list of policy modifiers below for how to change this.
+
WARNING: It is easy to get the bounds of a `squash` command wrong, with
confusing and destructive results. Beware thinking you can squash on a
selection set to merge all commits except the last one into the last
one; what you will actually do is to merge all of them to the first
commit _after_ the selected set.
+
Normally, any tag pointing to a combined commit will also be
pushed forward.  But see the list of policy modifiers below for how to
change this.
+
Following all operation moves, every one of the altered file
operation lists is reduced to a shortest normalized form. The
normalized form detects various combinations of modification,
deletion, and renaming and simplifies the operation sequence as much
as it can without losing any information.
+
The following modifiers change these policies:
+
|===================================================================
| `--delete`      | Simply discards all file ops and tags associated with deleted commit(s).
| `--no-coalesce` | Do not normalize the modified commit operations.
| `--pushback`    | Append fileops to parents, rather than prepending to children.
| `--pushforward` | Prepend fileops to children. This is the default; it can be
specified in a lift script for explicitness about intentions.
| `--tagforward`  | Any tag on the deleted commit is pushed
forward to the first child rather than being deleted. This is the
default; it can be specified for explicitness.
| `--tagback`     | Any tag on the deleted commit is pushed
backward to the first parent rather than being deleted.
| `--quiet`       | Suppresses warning messages about deletion of commits with
non-delete fileops.
| `--complain`    | The opposite of `--quiet`. Can be specified for explicitness.
| `--empty-only`  | Complain if a squash operation modifies a nonempty comment.
| `--blobs`       | Allow deletion of selected blobs.
|===================================================================
+
Under any of these policies except `--delete`,
deleting a commit that has children does not back out the changes made
by that commit, as they will still be present in the blobs attached to
versions past the end of the deletion set.  All a delete does when the
commit has children is lose the metadata information about when and by
who those changes were actually made; after the delete any such
changes will be attributed to the first undeleted children of the
deleted commits.  It is expected that this command will be useful
mainly for removing commits mechanically generated by repository
converters such as `cvs2svn`.

// COMMAND
include::docinclude/delete.adoc[]

// COMMAND
include::docinclude/rename.adoc[]

[[topology]]
=== Commit mutation

// COMMAND
include::docinclude/merge.adoc[]

// COMMAND
include::docinclude/unmerge.adoc[]

// COMMAND
include::docinclude/reparent.adoc[]

// COMMAND
include::docinclude/split.adoc[]

// COMMAND
include::docinclude/add.adoc[]

// COMMAND
include::docinclude/remove.adoc[]

// COMMAND
include::docinclude/tagify.adoc[]

// COMMAND
include::docinclude/reorder.adoc[]

[[branches]]
=== Advanced branch operations

// COMMAND
include::docinclude/branchlift.adoc[]

// COMMAND
include::docinclude/debranch.adoc[]

[[split_merge]]
=== Repository splitting and merging

// COMMAND
include::docinclude/divide.adoc[]

// COMMAND
include::docinclude/unite.adoc[]

// COMMAND
include::docinclude/graft.adoc[]

[[editing]]
=== Metadata editing

// COMMAND
include::docinclude/msgout.adoc[]

// COMMAND
include::docinclude/msgin.adoc[]

// COMMAND
include::docinclude/setfield.adoc[]

[[attribute_cmd,attribute]]
[SELECTION] attribute [ATTR-SELECTION] {show|set|delete|prepend|append} [ARG...]::
   Inspect, modify, add, and remove commit and tag attributions.
+
Attributions upon which to operate are selected in much the same way as
events are selected, as described in <<selections>>.
ATTR_SELECTION is an expression composed of
1-origin attribution-sequence numbers, '```$```' for last attribution,
'```..```' ranges, comma-separated items, '```(...)```' grouping,
set operations '```|```' union, '```&```' intersection, and '```~```' negation,
and function calls `@min()`, `@max()`, `@amp()`, `@pre()`, `@suc()`, `@srt()`.
Attributions can also be selected by visibility set '```=C```' for committers,
'```=A```' for authors, and '```=T```' for taggers.
Finally, '`/_regexp_/`' will attempt to match the regular
expression _regexp_ against an attribution name and email
address; '```/n```' limits the match to only the name, and '```/e```' to only
the email address.
+
With the exception of '```show```', all actions require an
explicit event selection upon which to operate. Available actions are:
+
[ATTR-SELECTION] show [>OUTFILE]:::
   Inspect the selected attributions of the specified events (commits and
   tags).  The '```show```' keyword is optional.  If no attribution
   selection expression is given, defaults to all attributions. If no event
   selection is specified, defaults to all events.  Supports > redirection.

{ATTR-SELECTION} set [NAME] [EMAIL] [DATE]:::
   Assign NAME, EMAIL, DATE to the selected attributions.  As a
   convenience, if only some fields need to be changed, the others can be
   omitted.  Arguments NAME, EMAIL, and DATE can be given in any
   order.

[ATTR-SELECTION] delete:::
   Delete the selected attributions.  As a convenience, deletes all authors
   if ATTR-SELECTION is not given.  It is an error to delete the
   mandatory committer and tagger attributions of commit and tag events,
   respectively.

[ATTR-SELECTION] prepend [NAME] [EMAIL] [DATE]:::
   Insert a new attribution before the first attribution named by
   ATTR_SELECTION.  The new attribution has the same type (_committer_,
   _author_, or _tagger_) as the one before which it is being inserted.
   Arguments NAME, EMAIL, and DATE can be given in any order.
+
If NAME is omitted, an attempt is made to infer it
from EMAIL by trying to match EMAIL
against an existing attribution of the event, with preference given to the
attribution before which the new attribution is being inserted.  Similarly,
EMAIL is inferred from an existing matching
NAME.  Likewise, for DATE.
+
As a convenience, if ATTR-SELECTION is empty or not
specified a new author is prepended to the author list.
+
It is presently an error to insert a new committer or tagger
attribution.  To change a committer or tagger, use "setfield"
instead.
+
[ATTR-SELECTION] append [NAME] [EMAIL] [DATE]:::
   Insert a new attribution after the last attribution named by
   ATTR_SELECTION.  The new attribution has the same type (_committer_,
   _author_, or _tagger_) as the one after which it is being inserted.
   Arguments NAME, EMAIL, and DATE can be given in any order.
+
If NAME is omitted, an attempt is made to infer it from EMAIL by
trying to match EMAIL against an existing attribution of the event,
with preference given to the attribution after which the new
attribution is being inserted.  Similarly, EMAIL is inferred from an
existing matching NAME.  Likewise, for DATE.
+
As a convenience, if ATTR-SELECTION is empty or not
specified a new author is appended to the author list.
+
It is presently an error to insert a new committer or tagger
attribution.  To change a committer or tagger, use "setfield"
instead.

// COMMAND
include::docinclude/prepend.adoc[]

// COMMAND
include::docinclude/append.adoc[]

// COMMAND
include::docinclude/gitify.adoc[]

// COMMAND
include::docinclude/filter.adoc[]

[[paths]]
=== Path reports and modifications

// COMMAND
include::docinclude/setperm.adoc[]

[[timequakes]]
=== Timequakes and time offsets

Modifying a repository so every commit in it has a unique
timestamp is often a useful thing to do, in order for every commit to
have a unique action stamp that can be referred to in surgical
commands.

The '```lint```' command will tell you if you have timestamp collisions.

// COMMAND
include::docinclude/timequake.adoc[]

// COMMAND
include::docinclude/timeoffset.adoc[]

Those of you twitchy about "rewriting history" should bear in
mind that the commit stamps in many older repositories were never very
reliable to begin with.
+
CVS in particular is notorious for shipping client-side
timestamps with timezone and DST issues (as opposed to UTC) that don't
necessary compare well with stamps from different clients of the same
CVS server. Thus, inducing a timequake in a CVS repo seldom produces
effects anywhere near as large as the measurement noise of the
repository's own timestamps.
+
Subversion was somewhat better about this, as commits were
stamped at the server, but older Subversion repositories often have
sections that predate the era of ubiquitous NTP time.

[[misc-surgical]]
=== Miscellanea

// COMMAND
include::docinclude/move.adoc[]

// COMMAND
include::docinclude/dedup.adoc[]

// COMMAND
include::docinclude/renumber.adoc[]

// COMMAND
include::docinclude/transcode.adoc[]

[[artifact-removal]]
== Artifact handling

Some commands automate fixing various kinds of artifacts
associated with repository conversions from older systems.

[[attributions]]
=== Attributions

// COMMAND
include::docinclude/authors.adoc[]

[[ignore]]
=== Ignore patterns

// COMMAND
include::docinclude/ignores.adoc[]

[[reference-lifting]]
=== Reference lifting

This group of commands is meant for fixing up references in commits
that are in the format of older version-control systems.  The general
workflow is this: first, go over the comment history and change all
old-fashioned commit references into machine-parseable reference cookies.  Then,
automatically turn the machine-parseable cookie into action stamps.
The point of dividing the process this way is that the first part is
hard for a machine to get right, while the second part is prone to errors
when a human does it.

Often Subversion references will be in the form 'r' followed by a
string of digits referring to a Subversion commit number. But not
always; humans come up with lots of ambiguous ways to write these.
CVS commit references are even harder to spot mechanically, as they're
just groups of digits separated by dots with no identifying prefix.

A Subversion cookie is a comment substring of the form
'```++[[SVN:ddddd]]++```' (example: '```++[[SVN:2355]]++```')
with the revision read directly
via the Subversion exporter, deduced from git-svn metadata, or
matching a `$Revision$` header embedded in blob data for the
filename.

A CVS cookie is a comment substring of the form
'```++[[CVS:filename:revision]]++```' or '```++[[CVS:revision]]++```' 
(example: '```++[[CVS:src/README:1.23]]++```'). The filename for the
revision-only form is deduced if the commit has exactly one fileop with
a path part

A mark cookie is of the form '```++[[:dddd]]++```' with d being
decimal digits, and is simply a reference to the specified mark. You
may want to hand-patch this in when one of the previous forms is
inconvenient.

An action stamp is an RFC3339 timestamp, followed by a '```!```',
followed by an author email address (author is preferred rather than
committer because that timestamp is not changed when a patch is
replayed on to a branch, but the code to make a stamp for a commit
will fall back to the committer if no author field is present).  It
attempts to refer to a commit without being VCS-specific.
Thus, instead of "```commit 304a53c2```" or "```r2355```",
"```++2011-10-25T15:11:09Z!fred@foonly.com++```".

The following git aliases allow git to work directly with action
stamps.  Append it to your _~/.gitconfig_; if you
already have an `[alias]` section, leave off the first line.

// Keep this updated from reposurgeon-git-aliases
--------
[alias]
	# git stamp <commit-ish> - print a reposurgeon-style action stamp
	stamp = show -s --format='%cI!%ce'

	# git scommit <stamp> <rev-list-args> - list most recent commit that matches <stamp>.
	# Must also specify a branch to search or --all, after these arguments.
	scommit = "!f(){ d=${1%%!*}; a=${1##*!}; arg=\"--until=$d -1\"; if [ $a != $1 ]; then arg=\"$arg --committer=$a\"; fi; shift; git rev-list $arg ${1:+\"$@\"}; }; f"

	# git scommits <stamp> <rev-list-args> - as above, but list all matching commits.
	scommits = "!f(){ d=${1%%!*}; a=${1##*!}; arg=\"--until=$d --after $d\"; if [ $a != $1 ]; then arg=\"$arg --committer=$a\"; fi; shift; git rev-list $arg ${1:+\"$@\"}; }; f"

	# git smaster <stamp> - list most recent commit on master that matches <stamp>.
	smaster = "!f(){ git scommit \"$1\" master --first-parent; }; f"
	smasters = "!f(){ git scommits \"$1\" master --first-parent; }; f"

	# git shs <stamp> - show the commits on master that match <stamp>.
	shs = "!f(){ stamp=$(git smasters $1); shift; git show ${stamp:?not found} $*; }; f"

	# git slog <stamp> <log-args> - start git log at <stamp> on master
	slog = "!f(){ stamp=$(git smaster $1); shift; git log ${stamp:?not found} $*; }; f"

	# git sco <stamp> - check out most recent commit on master that matches <stamp>.
	sco = "!f(){ stamp=$(git smaster $1); shift; git checkout ${stamp:?not found} $*; }; f"
--------

There is a rare case in which an action stamp will not refer uniquely
to one commit. It is theoretically possible that the same author might
check in revisions on different branches within the one-second
resolution of the timestamps in a fast-import stream.  There is
nothing to be done about this; tools using action stamps need to be
aware of the possibility and throw a warning when it occurs.

In order to support reference lifting, reposurgeon internally builds a
legacy-reference map that associates revision identifiers in older
version-control systems with commits.  The contents of this map come
from three places: (1) `cvs2svn:rev` properties if the repository was
read from a Subversion dump stream, (2) `$Id$` and `$Revision$` headers in
repository files, and (3) the _.git/cvs-revisions_
created by '```git cvsimport```'.

The detailed sequence for lifting possible references is this: first,
find possible CVS and Subversion references with the `=N` visibility
set; then replace them with equivalent delimited cookies; then run
`<<stampify_cmd>>` to turn the cookies into action stamps (using the
information in the legacy-reference map) without having to do the
lookup by hand.

// COMMAND
include::docinclude/stampify.adoc[]

It is not guaranteed that every such reference will be resolved,
or even that any at all will be. Normally all references in history
from a Subversion repository will resolve, but CVS references are less
likely to be resolvable.

// COMMAND
include::docinclude/legacy.adoc[]

A legacy-reference file maps reference cookies to (committer,
commit-date, sequence-number) triplets; these in turn (should) uniquely
identify a commit.  The format is two whitespace-separated fields:
the cookie followed by an action stamp identifying the commit.
+
It should not normally be necessary to use this command.  The
legacy map is automatically preserved through repository reads and
rebuilds, being stored in the file _legacy-map_ under
the repository subdirectory.

[[changelogs]]
=== Changelogs

CVS, Subversion, Mercurial, and Fossil do not have separated notions
of committer and author for changesets; when lifted to a VCS that
does, like git or bzr/brz, their one author field is used for both.

However, if the project used the FSF ChangeLog convention,
many changesets will include a ChangeLog modification listing an
author for the commit. In the common case that the changeset was
derived from a patch and committed by a project maintainer,
but the ChangeLog entry names the actual author, this information
can be recovered.

// COMMAND
include::docinclude/changelogs.adoc[]

The Co-Author convention described in the Linux kernel's
https://git.wiki.kernel.org/index.php/CommitMessageConventions[co-author message conventions]
is observed: If an attribution header is followed
by a whitespace-led line containing only a valid email address, that
name becomes the payload of a "Co-Author" header that is appended
to the change comment for the containing commit.

The command reports statistics on how many commits were altered.

[[coalescence]]
=== Clique coalescence

When lifting a history from a version-control system that lacks
changesets, it is useful to have a way to recognize cliques of
per-file changes that ought to be grouped into changesets.

You won't need this for CVS because cvs-fast-export does
clique coalescence itself.

// COMMAND
include::docinclude/coalesce.adoc[]

[[control-options]]
== Control Options

The following options change reposurgeon's behavior:

// TOPIC
include::docinclude/options.adoc[]

Most options are described in conjunction with the specific
operations that they modify.

Here are the commands to manipulate them.
None of these take a selection set:

// COMMAND
include::docinclude/set.adoc[]

// COMMAND
include::docinclude/clear.adoc[]

[[scripting-debugging]]
== Scripting and debugging support

[[macros]]
=== Variables, macros, and scripts

There are two different way to package command sequences for
re-execution; scripts and macros. Both are invoked by the same "do"
command. Scripts are the facility you can use use to build and store
conversion recipes as you develop them.

Occasionally you will need to issue a large number of complex surgical
commands of very similar form, and it's convenient to be able to
package that form as a text template so you don't need to do a lot of
error-prone typing.  For those occasions, reposurgeon supports simple
forms of named variables and macro expansion.

// COMMAND
include::docinclude/define.adoc[]

// COMMAND
include::docinclude/do.adoc[]

// COMMAND
include::docinclude/assign.adoc[]

// COMMAND
include::docinclude/unassign.adoc[]

// COMMAND
include::docinclude/undefine.adoc[]

// COMMAND
include::docinclude/print.adoc[]

Here are some more advanced examples of scripting:

--------
define lastchange {
@max(=B & [/ChangeLog/] & /%{0}/B)? list
}
--------

List the last commit that refers to a ChangeLog file containing
a specified string. (The trick here is that ? extends the singleton
set consisting of the last eligible ChangeLog blob to its set of
referring commits, and `list` only notices the
commits.)

--------
index >index.txt
shell <index.txt awk '/refs\/tags/ {print $4}' | sort | uniq | while read t; do echo "tag $(basename "$t") rename $(basename "$t" | sed -e 's/sample/example/')"; done >renames.script
script renames.script
--------

Mass-rename tags, replacing "sample" on the basename with
"example". Illustrates a general technique of generating reposurgeon
commands via shell that you then execute with the
'```<<script_cmd>>```' command. Enabling this technique is the reason
as many commands as possible support redirects.

[[housekeeping]]
=== Housekeeping

// COMMAND
include::docinclude/gc.adoc[]

[[diagnostics]]
=== Diagnostics

The destination of diagnostic logging is set by "set logfile"

// COMMAND
include::docinclude/log.adoc[]

[[debugging]]
=== Debugging

A few commands have been implemented primarily for debugging and
regression-testing purposes, but may be useful in unusual
circumstances.

// COMMAND
include::docinclude/resolve.adoc[]

// COMMAND
include::docinclude/version.adoc[]

// COMMAND
include::docinclude/hash.adoc[]

// COMMAND
include::docinclude/strip.adoc[]

[[profiling]]
=== Profiling

The "set readlimit" option can be used to limit processing to
head segments of an input stream file; this can be useful for
profiling and fault isolation.

Profiling commands are:

// COMMAND
include::docinclude/show.adoc[]

// COMMAND
include::docinclude/checkpoint.adoc[]

// COMMAND
include::docinclude/profile.adoc[]

// COMMAND
include::docinclude/exit.adoc[]

// END-TOC

[[subversion]]
== Working with Subversion (svn)

The transaction model of Subversion is nothing like that of the DVCSes
(distributed version-control systems) that followed it. Two of the
more obvious differences are around tags and branches. These
differences occasionally lead to conversion problems.

A Subversion tag isn't an annotation attached to a commit. The
Subversion data model is that a history is a sequence of surgical
operations on a tree; there are no annotation tags as such, a tag is
just another branch of the tree.  Accordingly a Subversion tag is a
copy of the state of an entire branch at a particular revision. This
can be losslessly translated to an annotation only if no additional
commits are added to the tag branch after the copy. But nothing
prevents this!  Reposurgeon tries to do the right thing, creating a
DVCS-style annotated tag using the metadata of the copy operation when
it can and otherwise preserving the changes as commits, using a
lightweight tag to point at the tip.

There is a subtler problem around branches themselves. In a
DVCS, deleting a branch removes it from the repository history
entirely, a fact of some significance since repositories are copied
around often enough that keeping every discarded experiment forever
would eventually drown the live content in superannuated cruft.
Subversion repositories, on the other hand, are designed on the
assumption that they sit on one server and never move.  A Subversion
branch is just a directory in the branch namespace; if you delete it,
you won't see it in following revisions, but if you update to an older
one that content will still be there. By default, reposurgeon will delete
the corresponding branches as if the deletion was done in a DVCS, keeping
only the commits that are also part of other branches' histories, but
you can tell it to preserve the branches instead and give them
unambiguous names in the `refs/deleted` namespace.

Bad things can happen when a tag directory is created, copied from,
deleted, then recreated from a different source directory. This is a
place where the Subversion model of tags clashes destructively with
the changeset-DAG model used by git and other DVCSes, especially if
the same tag is recreated later!  The obvious thing to do when
converting this sequence would be to just nuke the tag history from
the deletion back to its branch point, but that will cause problems if
a copy operation was ever sourced in the deleted branch (and this does
happen!).

What reposurgeon does instead is preserve the most recent branch with
any given name, so the view back from any branch tip in the repository
has the correct content. This does however mean that reposurgeon
discards the content of any previous branch having that same name.
However, see the `--preserve` option of the `read` command.

The reposurgeon analyzer tries to warn you about pathological cases,
and reposurgeon gives you tools for coping with them.  Unfortunately,
the warnings are (unavoidably) cryptic unless you understand
Subversion internals in detail.

There's another problem around Subversion merges. In a DVCS, a merge
normally coalesces two entire branches.  Subversion has something
close to this in newer versions; it's called a "sync merge" working on
directories (and is expressed as an svn:mergeinfo property of the
target directory that names the source).  A sync merge of a branch
directory into another branch directory behaves like a DVCS merge;
reposurgeon picks these up and translates them for you.

The older, more basic Subversion merge is per file and is expressed by
per-file svn:mergeinfo properties.  These correspond to what in
DVCS-land are called "cherry-picks", which just replay a commit from a
source branch onto a target branch but do *not* create cross-branch
links.

Sometimes Subversion developers use collections of per-file mergeinfo
properties to express partial branch merges.  This does not map to
the DVCS model at all well, and trying to promote these to full-branch
merges by hand is actually dangerous.  An excellent essay,
http://web.archive.org/web/20170104093332/https://plus.google.com/100357083629018071519/posts/jG7CN9R1SsZ[Partial git merges -- just say no],
explores the problem in depth.

The bottom line is that reposurgeon warns about per-file svn:mergeinfo
properties _and then discards them_ for good reasons.  If you feel an
urge to hand-edit in a branch merge based on these, do so with care
and check your work.

Three minor issues to watch for:

1. Superfluous root tags.

2. Legacy Subversion revision numbers.

3. File content mismatches due to $-keyword expansion.

More details follow.

=== Reading Subversion repositories

Note that the Subversion dump reader only supports versions 1 and
2 of the dump file format, not version 3 with diff-based file changes.
This shouldn't be a problem with normal use of reposurgeon, which
calls svnadmin dump in its default mode generating version 2.

If a stream file has gaps in the revision sequence, branch copy
references to nonexistent revisions will be patched to refer to the
most recent existing revision before the missing one.

Certain optional modifiers on the `<<read_cmd>>` command change
its behavior when reading Subversion repositories:

--user-ignores::
By default reposurgeon tosses out in-tree _.gitignore_ files found in the
history because they probably come from git-svn users who checked-in
their own _.gitignore_ files. Using this option makes reposurgeon
keep the content of these files and merge them with
the _.gitignore_ files generated from `svn:ignore` and `svn:global-ignores`
properties, if any.

--no-automatic-ignores::
Do not generate _.gitignore_ files from `svn:ignore` and
`svn:global-ignores` properties. If `--user-ignores` is also used
then only _.gitignore_ files that were present in the SVN tree will
exist in the final repository. If `--user-ignores` is not used,
no _.gitignore_ file at all will survive the conversion.

--preserve::
When a branch or tag was deleted in SVN, preserve the history up to
deletion in a git ref under `refs/deleted/`, instead of deleting the
branch and only keeping the commits that are also part of the history
of other branches.  The reference is disambiguated using the base
revision number of the dead branch. Also, preserve branch-copy commits
auto-generated by cvs2svn that would otherwise be discarded. (Note that
the reason --preserve is not the default behavior is because of
experience with large old repositories that may have hundreds or even
thousands of dead branches. While it is important that content copies
from dead branches be resolved correctly, the branches themselves are
almost never interesting.)

These modifiers can go anywhere in any order on the command line after
the `<<read_cmd>>` verb. They must be whitespace-separated.

As stacking up read options can result in a very long read invocation
line, it's useful to know that backslash is accepted as a continuation
character.

It is also possible to embed a magic comment in a Subversion stream
file to set these options (this is mainly useful in test
loads). Prefix a space-separated list of them with the magic comment
'``` # reposurgeon-read-options:```'; the leading space is
required. This may be useful when synthesizing test loads; in
particular, a stream file that does not set up a standard
`trunk/branches/tags` directory layout can use this to perform a
mapping of all commits onto the master branch that the git importer
will accept.

Here are the rules used for mapping subdirectories in a
Subversion repository to branches:

* '```trunk```' always becomes the master
branch.  Directories under '```branches```' become gitspace branches;
directories under '```tags```' become gitspace tags.

* Each potential tag is checked to see if it has commits on it
after the initial creation or copy.  If there are such commits, or if the
branch creation or copy introduces changes other than the copy, it
becomes a branch.  If not, it may become a tag in order to preserve
the commit metadata. In all cases, the name of any created tag or
branch is the basename of the directory.

* Files in the top-level directory are assigned to a synthetic branch
named '```unbranched```'. If the Subversion repository has a branch
named '```unbranched```' the name '```unbranched-bis```' is used
instead; actually, '```-bis```' is appended enough times to get to an
unused branch name.

Branch-creation operations with no following commits are usually
tagified. However, this is done to preserve comment/committer
data entered by users; when reposurgeon can detect that a
branch-creation comment was automatically generated (as often
happens in cvs2svn conversions) the commit will simply be discarded
so as not to create clutter that has to be manually removed by the
operator. (That discard action is prevented by the --preserve option.)

Otherwise, each commit that only creates or deletes directories
(in particular, copy commits for tags and branches, and commits that
only change properties) will be transformed into a tag named after the
tag or branch, containing the date/author/comment metadata from the
commit.

Subversion branch deletions are turned into deletealls, clearing the
fileset of the import-stream branch.  When a branch
finishes with a deleteall at its tip, the deleteall is transformed
into a tag. This rule cleans up after aborted branch renames.

Occasionally (and usually by mistake) a branchy Subversion repository
will contain revisions that touch multiple branches. These are handled
by partitioning them into multiple import-stream commits, one on each
affected branch. The Legacy-ID of such a split commit will have a
1-origin split number separated by a dash - for example, if Subversion
revision 2317 touches three branches, the three generated commits will
have IDs 2317-1, 2317-2, and 2317-3.

The `svn:executable` and `svn:special` properties are translated
into permission settings in the input stream; `svn:executable` becomes
100755 and `svn:special` becomes 120000 (indicating a symlink; the blob
contents will be the path to which the symlink should resolve).

Any `cvs2svn:rev` properties generated by
`cvs2svn` are incorporated into the internal map
used for reference-lifting, then discarded.

Normally, per-directory `svn:ignore` properties (and
`svn:global-ignores` properties, e.g. in a site configuration file)
become _.gitignore_ files.  Actual _.gitignore_ files in a Subversion
directory are presumed to have been created by git-svn users
separately from native Subversion ignore properties and discarded with
a warning. It is up to the user to merge the content of such files
into the target repository by hand.  But this behavior is changed by
the `--user-ignores` option which disables filtering of in-tree
_.gitignore_ files and instead merges them with _.gitignore_ files
generated from Subversion properties. On the other hand, the
`--no-automatic-ignores` option discards
Subversion `svn:ignore` and `svn:global-ignores` properties without translation.

Any _.cvsignore_ files left over from a Subversion repository's
ancient history as a CVS repository are deleted. Instead the
(presumably more up-to-date) Subversion ignore properties are
translated.

`svn:mergeinfo` properties are interpreted.  Any `svn:mergeinfo` property on a
revision A with a merge source containing all revisions on a branch from the
forking point (or the branch start if the histories are independent) up to
revision B produces a merge link such that the branch tip at revision B becomes
a parent of A.
The "svnmerge-integrated" properties produced by Subversion's svnmerge.py script
are handled the same way.

All other Subversion properties are discarded. (This may change in a
future release.) The property for which this is most likely to cause
semantic problems is `svn:eol-style`. However, since property-change-only
commits get turned into annotated tags, the translated tags will retain
information about setting changes.

The sub-second resolution on Subversion commit dates is discarded;
Git wants integer timestamps only. Normally Subversion timestamps are
rounded down, but when two adjacent timestamps have the same seconds
part and the later one is in the top half-second of the interval,
the later one is rounded up instead.  This does much to reduce
collisions while guaranteeing that no timestamp is ever shifted
to a non-adjacent second mark.

Because fast-import format cannot represent an empty directory,
empty directories in Subversion repositories will be lost in
translation.

Subversion local usernames are mapped in the style of `git
cvs-import`; user '```foo```' becomes '```foo <foo>```', which is
sufficient to pacify git and other systems that require email
addresses. You can remap them to real addresses using the
`<<authors_cmd>> read` command.

Reading a Subversion stream enables writing of the legacy map as
'legacy-id' passthroughs when the repo is written to a stream
file.

Reposurgeon tries hard to silently do the right thing, but there are
Subversion edge cases in which it emits warnings because a human may
need to intervene and perform fixups by hand. Here are the less
obvious messages it may emit:

properties set::
   reposurgeon has detected a setting of a user-defined property, or
   the Subversion properties `svn:externals`. These properties cannot
   be expressed in an import stream; the user is notified in case this
   is a showstopper for the conversion or some corrective action is
   required, but normally this error can be ignored.  This warning is
   suppressed by the `--ignore-properties` option.

Detected link from <__revision__> to <__revision__> might be dubious::
   When trying to detect parent links from multiple file copies like what
   `cvs2svn` can produce, source revisions of the different copies were not
   all the same. The link should probably be monitored because it has a
   non-negligible probability of being slightly wrong. This does not impact
   the tree contents, only the quality of the history.

[[mid_branch]]
=== Mid-branch deletions

When a branch A is deleted and a branch B is copied to the name A, the
Subversion intent is to replace the contents of branch A with the
contents of branch B, keeping the A name.  This is a poor man's merge
from before "svn merge" existed. Many Subversion users who formed
their habits before svn merge existed still operate this way.

In git terms, this almost corresponds to a merge of A into B
followed by a rename of B to A.  Branch B continues to exist, however,
so we can't do that in translation.  The
reposurgeon logic does not try to be clever about this,
because "clever" would have rebarbative edge cases; the sequence is
translated into a deleteall followed by a commit operation that
recreates the B files under corresponding A names. No merge link is
created. The commit filling A with a branch copy from B will have
B as its first parent, though, so all that would be needed is to create
a merge link from the old A before the delete to the commit recreating A.

This case is mentioned here because it is likely to confuse
the merge-tracking algorithms used, e.g., by `git diff`, or if you ever
try to merge a branch that forked off the old A to a branch spun off the new
(and expect git to know that you do not want to incorporate old A's changes).

[[multiproject]]
=== Multiproject Subversion repositories

By convention, subversion repository is supposed to have a regular
structure, what we'll call "standard layout".  This has three
top-level directories, trunk/ and tags/ and branches/. In standard
layout, directories beneath tags are never modified after the copy that
creates them.

A repository in standard layout can be converted nearly losslessly into
Git and other modern DVCSes, with minor exceptions around deleted and
later recreated branches and some Subversion-specific file properties.

But standard layout is only a convention; the Subversion tools don't
actually enforce it.  One very simple non-standard layout is a "flat"
repository.  We'll discuss these in detail in a <<flat,later
section>>.

In a more common non-standard layout, Subversion repositories are
organized to hold multiple projects, with the root directory
containing one subdirectory per project and each project subdirectory
having a standard layout underneath it.

We'll call this kind of layout "conformable" when it is perfectly
regular, not containing any of the following exceptions:

* Top-level project directories that _don't_ have a standard
layout underneath

* Copy operations on top-level subdirectories.

* trunk/tags/branches structure more than one level beneath
the top level.

These sorts of irregularities do not fit well into reposurgeon's
internal data model. The fallback behaviors in reposurgeon that
attempt to handle them are fragile and kludgy.  Attempting to read
such a stream may confuse or even crash reposurgeon's reader for
Subversion dumps.

Unfortunately, it is rare for a multiproject repository not to have
defects of this kind. There's an easy way to check; feed the stream to
repocutter swapcheck.  If you get an empty report, the stream is
perfectly conformable; otherwise, each report line flags a defect
you need to fix or remove.  Look at the output of "repocutter help
swapcheck" for more details.

Conversions of non-conformable multiproject Subversion repositories
are messy and tricky.  If you work for an organization with a budget,
you should seriously consider engaging the author to do yours - that's
likely to save you time and money relative to what you'd pay someone
in-house to climb the learning curve.

The general strategy in these cases has three steps:

. Transform a stream dump from the repository into a fully-conformable
  one.

. Apply repocutter swapsvn to transform that stream to one
  with standard layout.

. Load the standard-layout stream into reposurgeon and trim out
  unneeded content inside reposurgeon.

The first step is the tricky one. There are two general tactics you
can pursue towards this: (1) expunging badly-structured
content, and/or (2) using repocutter pathrename to massage
non-conformable directories into conformable ones.

Before you think about doing expunges, be aware that they can be
complicated by cross-project copy operations.  If you delete a
directory that's ever a source for a copy into conformable structure,
havoc will ensue; a very common case of this is copies from obsolete
branches to trunk.  That's why it's generally best to defer trimming
unwanted history until you have it inside reposurgeon.

Generally you're going to do the heavy lifting with repocutter
pathrename operations. The author has seen conversions that literally
required hundreds of these.  You may also need to do repocutter
filecopy and skipcopy operations - but here be dragons; by the time
you need to do those you should reconsider not hiring the author.

It cannot be emphasized enough that conversions of non-conformable
multi-project repositories are *difficult*; expect your timeframe to
completion to be measured not in hours or days but months. Improving
to tools to make these less painful is the reposurgeon project's main
area of ongoing research.

==== Some caveats about repocutter swapsvn

All project-local tags and branches will be promoted to become tags
and branches across the entire repository.

Paths that swapsvn does not recognize as conformable will be passed
through unaltered.  Don't try to treat this as a feature;
reposurgeon's fallbacks for dealing with these are crude and fragile.

Subversion mergeinfos, being file- and directory-oriented rather than
true branch merges, often can't be carried over even in a
single-project conversion. The global branch coalescence that
repocutter swapsvn does multiplies the problem; you may simply have to
discard those properties to get a clean conversion.

See the embedded help listed by "repocutter help swapsvn" for some
details and caveats.

[[flat]]
=== Flat repositories

If the repository has no branch structure at all, you might want to
filter the stream with "repocutter push trunk" and then individually
turning subdirectories into branches using the <<branchlift_cmd>>
command.  The pattern for that looks like this:

--------
read <multiproject.svn
branchlift master branches/release-1.0
branchlift master branches/release-2.0
rename path "trunk/(.*)" "${1}"
--------

This will turn two subdirectories into branches named "release-1.0"
and "release-2.0", then pops "trunk" off all the paths where it occurs
(leaving those commits on the master branch).

This method has the disadvantage that you have to enumerate all
branches you want to lift. Still, it may be useful on repositories
that consist of one big unbranched file tree not conforming to a
standard layout.

If for some reason you want to treat a branchy repository as though
it's flat, using repocutter to rename the "trunk" directory to
something else first (like, say, "main") will foil the logic in the
reposurgeon stream reader that wants to do branch and tag analysis.

[[CVS]]
== Working with CVS (cvs)

When you are converting a CVS repository using reposurgeon, most of the
heavy lifting will have been done by the importer - cvs-fast-export.
In particular, it coalesces CVS per-file changes into changesets when
it detects that they have identical comments and attributions and are
close in time, and it converts .cvsignore files to .gitignores.

A CVS repository normally consists of a set of module subdirectories
and a CVSROOT directory containing metadata. cvs-fast-export ignores
CVSROOT; thus you can run reposurgeon at any level of a directory tree
containing CVS master files, and it will try to lift what it can see at
and below the current directory it is run from.

If you do this at the top level of the repository directory, your
converted repository will have a subdirectory corresponding to each
module. This is normally not the way you want to do things, as CVS
tags are not likely to be consistent across all modules and thus won't
lift correctly. You probably want to do individual module conversions.

Problems in CVS conversions generally arise from the fact that CVS's
data model doesn't have real multi-file changesets, which are the
fundamental unit of a commit in DVCSes.  It can be difficult to fully
recover changesets from what are actually large numbers of single-file
changes flying in loose formation - in fact, old CVS operator errors
can sometimes make it impossible. Bad tools silently propagate such
damage forward into your translation. Good tools, like cvs-fast-export
and reposurgeon, warn you of problems and help you recover.

Here are the kinds of conversion glitches to watch for:

1. Failure to coalesce runs of comments with identical attribution
and comment text.

2. Superfluous root tags.

3. Legacy CVS revision numbers.

4. File content mismatches due to $-keyword expansion.

5. "Zombie" files due to failure to track deletion operations.

Details follow.

Glitch #1 Is driven by whether the window defining "close in
time" is wide enough.  If it's not, you may detect commit groups with
the same committer and comment text that should have been merged into
one changeset but were not.  You can either clean these up with the
'```<<coalesce_cmd>>```' command in reposurgeon or run cvs-fast-export
by hand with a larger `-w` option and read in the generated stream.

Glitch #2: In cleaning up a CVS conversion that is unique to that
system is deleting root tags - tags which have "-root" as a name
suffix and mark the beginning of a branch,  CVS uses these for
bookkeeping, but later systems don't need them.  They're just
clutter and can be removed.

Glitch #3: It's also worth paying careful attention to
reference-lifting so that you can scrub useless CVS revision numbers
out of comments.  This is a more pressing issue than it is with
Subversion, where changesets map to changesets, and conversions have
the option of marking each target changeset with its revision number.

Glitch #4: You can spot content mismatches due to keyword expansion easily.  They
will produce single-line diffs of lines containing dollar signs
surrounding keyword text.  Because binary files can be corrupted by
keyword expansion, cvs-fast-export behaves like cvs -kb mode and does
no keyword expansion of its own.

[[zombie]]
Glitch #5: Manifest mismatches on tags are most likely to occur on
files which were deleted in CVS but persist under later tags in the
Git conversion.  You can bet this is what's going on if, when you
search for the pathname in the CVS repository, you find it
in an Attic directory.

These spurious reports happen because CVS does not always retain
enough information to track deletions reliably and is somewhat flaky
in its handling of "dead"-state revisions. To make your CVS and git
repos match perfectly, you may need to add delete fileops to the
conversion - or, more likely, move existing ones back along their
branches to commits that predate the gitspace tag - using
reposurgeon.

Manifest mismatches in the other direction (present in CVS, absent
in gitspace) should never occur.  If one does, submit a bug report.

Any other kind of content or manifest match - but especially any on
the master branch - is bad news and indicates either a severe
repository malformation or a bug in cvs-fast-export (or possibly
both). Any such situation should be reported as a bug.

Conversion bugs are disproportionately likely to occur on older
branches or tags made with CVS version before CVS got commit IDs in
2006 (version 2.12). Often the most efficient remedy is simply to
delete junk branches and tags; reposurgeon(1) makes this easy to do.

[[mercurial]]
== Working with Mercurial (hg)
There is a built-in extractor class to perform extractions from
hg repositories.

There are some important caveats about hg. Please read this section
carefully before attempting conversion or surgery.

In hg the "branch" of a commit is a name carried in the commit
and inherited by any child of it that is created - it behaves
like a color on the commit rather than being implied by an ancestry
relationship to a named branch tip (as in git) or by a location
in a directory layout (as in svn).

When different people add commits to a branch, and then
push their results to the same upstream repository, each person's
commits become different commit sequences, each attached to wherever the
branch head was at the time they last pulled. So a branch
can have multiple heads.  The heads themselves do not have names,
other than the hex IDs of their tip commits.

Typically someone will then merge the changes from all the heads into
following commits on their branch.  There is an
https://wiki.mercurial-scm.org/BranchingExplained[illuminating
diagram] of this.

Before importing a Mercurial repository, *you should merge-resolve
all branches so they have single heads* first. The extractor can't
handle multiple-headed branches gracefully.  This is because
reposurgeon internally uses a data model similar to git's, but
enriching that model wouldn't help much because no other VCS
has an analog multi-headed branches.

In hg branches can have a "closed" status marking that line of
development as terminated. This flag is not represented in reposurgeon
and can't be exported to other VCSes.

/////////////////////////////////////////////////////
Date: Wed, 20 Dec 2023 01:53:58 +0000
From ec429@cantab.net  Tue Dec 19 21:03:27 2023
From: Edward Cree <ec429@cantab.net>
To: "Eric S. Raymond" <esr@thyrsus.com>
Subject: Re: How does the extractor handle branches with multiple heads?

On 20/12/2023 00:42, Eric S. Raymond wrote:
> You seem to be the person who contributed actual knowledge of
> Mercurial to reposurgeon's Mercurial extractor class. So I
> have a question.
>
> What does it do with Mercurial branches that have multiple heads?

If I ever knew this for certain I've long since forgotten, sorry.
My best guess, based on re-reading the code and a few tests in a
toy Hg repo, is as follows:
Every commit will get branch coloured according to which branch
Hg says it belongs to.
refs/heads/ will contain only one head for each branch, whichever
one 'hg branches' outputs (I think this is the most recent head
on the branch in the linear local ordering, which is horrifically
arbitrary but hey that's Mercurial).
Consequently the other heads will not be reachable from any
branch ref; user will probably want to create new branches from
them otherwise they'll get GCed by git.

If you wanted to improve this I guess the starting point would be
to run
    hg heads -T "{node|short}\t{branch}\n"
to generate a list of all heads and the branches they're on, and
then create disambiguated branch names so you could give every
one of them a ref.

I don't think I considered this issue at the time because getting
multiple heads into your central Hg repo (rather than individual
developers' clones) takes some especially bad practice, and it
should always be possible to fix it in Hg with a dummy merge.
/////////////////////////////////////////////////////

There is another feature in hg that resembles named branches:
bookmarks - symbolic names for branch-tip commits that you assign, which
won't be identical to the name of the branch you're on unless you set
them that way, and which move to the new tip on the branch every time
you add one. There's a https://wiki.mercurial-scm.org/Bookmarks[detailed
description].

By default, bookmarks are ignored in favor of the name of the branch
they are on. You can specify explicit handling for bookmarks by
setting '```reposurgeon.bookmarks```' in your _.hg/hgrc_. Set the
value to the prefix that reposurgeon should use for bookmarks.

For example, if your bookmarks represent branches, put this at
the bottom of your _.hg/hgrc_:

--------
[reposurgeon]
bookmarks=heads/
--------

If you do that, it's your responsibility to ensure that branch names
do not conflict with bookmark names. You can add a prefix like
'```bookmarks=heads/feature-```' to disambiguate as necessary.

Mercurial tags are name-to-commit mappings stored in a (versioned)
dotfile in your repository directory. They are exported to other
VCSes as lightweight tags.

[[hg-subrepo]]
The hg extractor does not attempt to recursively handle subrepos.  Rather,
it will extract the history of the top-level repo, in which _.hgsub_ and
_.hgsubstate_ will be treated as regular files.  If you wish to translate
these into the semantics of your target VCS, you will need to do so with
surgical primitives after reading the history into reposurgeon.

[[tipsntricks]]
== Tips and Tricks

=== Use the Q set to check your work

When you're experimenting with mass changes based on pattern matching
(as in filter, transcode, changelogs, or even msgin) bear in mind that
commits and other objects actually modified get their Q bits set. You
can use this as a selection with "list" commands to check your work.

=== Pushing and popping pathname segments

Because "/" is the default delimiter for regular expressions, it may
not be obvious how to write commands that have to manipulate
pathnames by segment.  Here is how to handle some common cases:

-----------------------------------
# Push a pathname segment onto filenames
rename path /^(.*)/ "foo/${1}"

# Pop a pathname segment off filenames
rename path :foo/(.*): "${1}"
-----------------------------------

The second recipe uses the fact that almost any delimiter character
will actually do. But note that you can't use single quote; that is
interpreted as instruction to match a literal string rather than a
regexp.

You also can't use double quotes, as those are striped off argument
tokens before they are passed to the functions that interpret them.
Thus , in the following command...

-----------------------------------
rename path "foo/(.*)" "${1}"
-----------------------------------

...the first argument would be interpreted as a literal path name and
not a regexp. However, this would work as probably intended:

-----------------------------------
rename path ":foo/(.*):" "${1}"
-----------------------------------

That is, the double quotes will be stripped off at argument parsing
time and then the matching colons will cause the first argument to be
recognized as a delimited regexp. This will be useful if you need to
specify a delimited regexp that contains whitespace; you can
string-quote it so the argument parser will pass it as a single token.

=== Extracting the history of a single file

You can use a negative selection to extract the history of a single
file. Using the reposurgeon repository itself as an example:

-----------------------------------
read .
clone
delete path --not ci/prepare.sh
rename path :ci/(.*): "${1}"
-----------------------------------

After thus commands, the reposurgeon-clone repository will
contain only the history of ci/prepare.sh, with the "ci" directory
popped off the filename in the way indicated in the previous tip.

=== Use macros to avoid repeating boilerplate

We can encapsulate that technique for stripping off a directory as
follows:

-----------------------------------
define pop rename path %{0}/(.*): "${1}"
-----------------------------------

You could then say:

-----------------------------------
do pop junkdir
-----------------------------------

Here's a less trivial example:
In CVS repositories of projects that use the GNU ChangeLog convention,
a very common pre-conversion artifact is a commit with the comment
"```++*** empty log message ***++```" that modifies only a ChangeLog
entry explaining the commit immediately previous to it. The following

--------
define changelog <%{0}> & /empty log message/ squash --pushback
do changelog 2012-08-14T21:51:35Z
do changelog 2012-08-08T22:52:14Z
do changelog 2012-08-07T04:48:26Z
do changelog 2012-08-08T07:19:09Z
do changelog 2012-07-28T18:40:10Z
--------

is equivalent to the more verbose

--------
<2012-08-14T21:51:35Z> & /empty log message/ squash --pushback
<2012-08-08T22:52:14Z> & /empty log message/ squash --pushback
<2012-08-07T04:48:26Z> & /empty log message/ squash --pushback
<2012-08-08T07:19:09Z> & /empty log message/ squash --pushback
<2012-07-28T18:40:10Z> & /empty log message/ squash --pushback
--------

but you are less likely to make difficult-to-notice errors typing the
first version.

(Also note how the text regexp acts as a failsafe against the
possibility of typing a wrong date that doesn't refer to a commit with
an empty comment. This was a real-world example from the CVS-to-git
conversion of groff.)

=== Handling username collisions

Occasionally when processing a CVS or Subversion repository you will
encounter a case where, during two different spans of time, the
same user ID was used by two different people for which you have
full name-and-email attributions you want to apply.

As an example, suppose your history has two committers named
"fred", one active before March 15 2010 and one after.  The name and
email fields in their commit attributions all look like "fred <fred>"
with variable data following.

There's no way you can discriminate by time in an authors file.
But there's a different technique that will work.  Like this:

----
1..<2010-90-15T00:00:00Z> & /^fred$/C attribute =C|=A set "Fred Foonly" "fred1@foonly.org"
<2010-90-15T00:00:00Z>..$ & /^fred$/C attribute =C|=A set "Fred J. Muggs" "fred2@foonly.org"
/fred[12]/C list inspect
----

The third command allows you to check that the previous two produced
a sane-looking result.

Part of the trick here is knowing that if you only set name and email
fields with an attribute command the date field is left unchanged.

If you needed to handle more than one of these collisions, you
might consider a slightly different approach that illustrates
how to write and use a multi-line macro:

----
define splitAttribution {
1..<{1}> & /^%{0}$/C attribute =C|=A set "%{2}" "%{2}"
<{1}>..$ & /^%{0}$/C attribute =C|=A set "%{3}" "%{3}"
}
do splitAttribution fred 2010-90-15T00:00:00Z fred1 fred2
----

This depends on the fact that the the script arguments %{0} and %{1} get
substituted before the regular-expression compiler interprets {}.
You'd follow it by applying an attribution file containing entries
for fred1 and fred2.

=== Use reposurgeon.el to cut down on repetitive handwork

An Emacs mode for editing msgout dumps ships with the reposurgeon
distribution.  It includes various useful commands for Gitifying
comments and reference-lifting.

=== Avoid git-svn lossage with repocutter filtering

Subversion histories that have been touched by git-svn often have
subtle damage in the metadata.  One common class of problems comes from
git-svn creating .gitignore and .git histories in the Subversion
repository. Commits on these paths are best treated as junk and removed.

While they could be removed with a reposurgeon delete path command after
the repository has been translated to Git, sometimes one sees a worse
problem resulting from git-svn somehow creating invalid copy nodes
involving .git/.gitignore.  This can produce alarming log messages,
or even cause a Subversion stream read to fail spuriously despite the
actual Subversion part of the history being undamaged.

If you suspect this is happening, the following repocutter expunge
command is your friend:

----
repocutter expunge "/.git$"  "/.gitignore$"
----

If you use this to pre-filter your repository dump it will remove the
garbled copy nodes along with every other modification to matching
path, without affecting anything in the actual native Subversion
part of the history.

=== Prevent timestamp collisions with timequake

If you're going to use msgout/msgin to batch-edit commit metadata,
consider running the timequake command first to ensure that all commit
timestamps are unique. This way it will be guaranteed that you can feed
the output of msgout --id to msgin without problems due to matching
multiple commits.

After running timequake it's a good idea to do "=Q list" to check
that it hasn't had surprising effects.

[[troubleshooting]]
== Troubleshooting and bug reports

[[reposurgeon-crashes]]
=== When reposurgeon crashes

If reposurgeon crashes on a Subversion stream dump that has not been
altered or filtered in any way since being emitted by svnadmin dump,
that is definitely a bug; please <<reporting-bugs,report it as such>>.
To speed up the fix, follow the instructions on preparing a reduced
test case.

You are more likely to see a crash when processing a stream that has
been modified by filtering out unwanted commits, e.g. via select or
expunge or sift commands.  The most common problem is copyfrom
operations referring to a source revision that has been removed.
In theory reposurgeon should fall back to referencing the latest
revision before the gap, but there are still occasional crashes
the the reposurgeon maintainers have not successfully characterized.

[[oom]]
=== Dealing with memory exhaustion

To do its job, reposurgeon needs to hold all of your history's metadata
in memory. That doesn't mean the content part, but does mean all of
the changeset attributions, comments, and tags.  Given a large enough
repository, this will overrun the RAM of a small machine. If this
happens to you, your reposurgeon instance will die abruptly with an
OOM (Out Of Memory) error while attempting to read in your repository.

It is extremely unlikely that this is due to a bug in reposurgeon.
Before filing an issue about it, there is a procedure you should try.
It consists of bisecting on the parameter the Go language runtime
uses to control the frequency of garbage collection.  You can set this
using the environment variable GOGC or reposurgeon "<<gc_cmd>>" command.

GOGC defaults to 100, which instructs the runtime to garbage-collect
when the heap size is 100% bigger than (i.e., twice) the size of the
reachable objects after the last garbage collection.  To increase the
frequency of GC, usually resulting in a lower memory high-water mark,
decrease that percentage threshold.  To decrease gc frequency, increase
the threshold so the runtime tolerates a larger heap.

To troubleshoot your OOM problem, bisect on this threshold to find
the highest value that will avoid OOM.  Start by cutting it to 50,
then to 25, then to 12, then to 6.  If you find a value that allows
you to read to completion, you may want to try increasing it by
a half interval (e,g. 50 to 75, 25 to 37, etc.) to get back some
throughput.

If your repository won't read in at GOGC=6 you have a real problem.
Unfortunately, it's not one the reposurgeon devteam can help you with;
the correct solution to it is to do your conversion on a machine with
more RAM and/or more swap configured.  64GB should be sufficient.  The
largest repository the reposurgeon devteam has ever seen (the history
of GCC, 280K Subversion commits) fit on a 128GB machine with GOGC=30.

If you can't read your history onto a 128GB machine with GOGC=30,
*then* maybe the reposurgeon devteam ought to hear about it. That
said, if you can find ways to make reposurgeon more efficient, we are
eager to accept those patches, or even just a bug report with the
details. It's probable that there are some efficiency gains yet to be
made.

[[stallout]]
=== Dealing with stalled conversions

Occasionally it will happen that a conversion on a particularly large
or malformed repository seems to stall out, grinding endlessly without
completing a conversion phase.

Reposurgeon's execution time is dominated by cycles spent in the
memory allocator and garbage collector.  Thus, you can pay RAM to
decrease running time - push GOGC *up* from its default of 100.  If
your conversion completes in reasonable time before your memory usage
increases enough that reposurgeon gets killed by OOM, you win.
Otherwise see the previous section about adding RAM and swap space.

A stallout is more difficult to troubleshoot than an OOM, and more likely
to indicate an actual bug or algorithmic problem in reposurgeon. There
are a couple of things you can do to make a good resolution more
likely:

1. Identify and report the phase in which the stallout occurs.  Be
   aware that there is a known problem that phase C2 of the
   Subversion dump reader that really thrashes the allocator; that's
   not reducible and we're just going to tell you you have to throw
   more RAM and compute clocks at the problem.

2. Use repobench to see if you can identify a revision that triggers
   the stall.  The procedure for this is to use it to step your
   readlimit value up from zero until you see the runtime spike.

3. As always, provide a <<repocutter.adoc#,stripped>> (and possibly
   <<repocutter.adoc#,obscured>>) dump of the repository for testing.

[[reporting-bugs]]
=== How to report bugs

Be aware that bugs can be due to problems in any of (a) reposurgeon
itself, (b) the front end it's using to read your repository, or (c)
the back end that writes the modified repository.  For example, when
you have a bug in a CVS conversion it is more than likely to be a
problem with the cvs-fast-export front end.  In that case you need to
read
https://gitlab.com/esr/cvs-fast-export/-/blob/master/reporting-bugs.adoc[Reporting
bugs in cvs-fast-export] and follow those directions.

It is generally not possible to reproduce reposurgeon/repocutter bugs without a
copy of the history on which they occurred. When you find a bug,
please send the maintainers:

(a) An exact description of how observed behavior differed from expected
behavior.  If reposurgeon/repocutter died, include the backtrace.

(b) A git fast-import or Subversion dump file of the repository you
were operating on, or in the case of CVS the whole repository.
Alternatively a pointer to where it can be pulled from - though a
static copy submitted with the bug is better is better because commits
added afterwards can confuse diagnostics.

(c) A script containing the sequence of reposurgeon or repocutter commands that
revealed the bug.  If you were exploring interactively, remember
that the "<<history_cmd>>" command exists and can dump your command history
to a file.

(d) If you were using the standard-workflow Makefile generated by
"<<repotool.adoc#,repotool>> initmake", mention that in your bug
report.  If you modified the Makefile, include a copy with the bug
report.

Please use the reposurgeon project's issue tracker and attach these
files. It's helpful.

Are you seeing git die with a complaint about an unknown
--show-original-IDs option?  Upgrade your git; reposurgeon
needs 1.19.2 or later.

==== Test case reduction

If you know how to reproduce the error, the best possible test case
is a hand-crafted dump stream of minimal size with content that
explains how it breaks the tool. Those are turned into regression tests
instantly.

When you don't know the cause of the error, ship the project a dump
file derived from the real repository and a script containing the
commands that triggered it.  To speed up the debugging process so you
can get an answer more quickly, there are some tactics you can use to
reduce the bulk of the test case you send.

How to make dumps in Git: cd to your git repository and capture the
output of "<<repotool.adoc#,repotool>> export".

How to make dumps in Subversion: cd to the toplevel directory of the
repository master - not a checkout of it. You can tell you're
in the right place if you see this:

----------------------------------------------------------------------
$ ls
conf  db  format  hooks  locks  README.txt
----------------------------------------------------------------------

Then run "<<repotool.adoc#,repotool>> export", capturing the output.

The commands you will use for test-case reduction are reposurgeon and,
on Subversion dumps, <<repocutter.adoc#,repocutter>>.

There is a cvsstrip tool in the cvs-fast-export distribution
that performs an analogous stripping operation on CVS repositories.
Please see the troubleshooting instructions included with that
distribution to lean how to use it.

==== Replace the content blobs in the dump with stubs

The subcommand in both tools is 'strip'; it will usually cut the size
of the dump by more than a factor of 10. Check that the bug still
reproduces on the stripped dump; if it doesn't, that would be
unprecedented and interesting in itself.

If you are trying to maintain confidentiality about your code, sending
me a stripped repo has the additional advantage that the code won't be
disclosed!  The command preserves structure and metadata but replaces
each content blob with a unique magic cookie.

If you don't want to disclose even the metadata, you can do a
<<repocutter.adoc#,repocutter>> "obscure" pass after the strip. This
will mask file paths and developer names.

==== Truncate the dump as much as possible

Try to truncate the dump to the shortest leading section that
reproduces the bug.

A reposurgeon error message will normally include a mark, event
number, or (when reading a Subversion dump) a Subversion revision
number.  Use a selection-set argument to reposurgeon's '<<write_cmd>>'
command, or the 'select' subcommand of
<<repocutter.adoc#,repocutter>>, to pare down the dump so that it ends
just after the error is triggered.  Again, check to ensure that the
bug reproduces on the truncated dump.

If the error message doesn't tell you where the problem occurred, try
a bisection process.  Use the readlimit option to ignore the last half
of the events in the dump; check to see if the bug reproduces.  If it
does, repeat; if it does not, throw out the last quarter, then the
last eighth, and so forth.  Keep this up until you can no longer drop
events without making the bug go away.

Bisection is more effective than you might expect, because the kinds
of repository malformations that trigger misbehavior from reposurgeon
tend to rise in frequency as you go back in time. The largest single
category of them has been ancient cruft produced by cvs2svn conversions.

==== Topologically reduce the dump

Next, topologically reduce the dump, throwing out boring commits that are
unlikely to be related to your problem.

If a commit has all file modifications (no additions or deletions or
copies or renames) and has exactly one ancestor and one descendant,
then it may be boring. In a Subversion dump it also has to not have
any property changes; in a git dump it must not be referred to by any
tag or reset.  Interesting commits are not boring, or have a
not-boring parent or not-boring child.

Try using the 'reduce' subcommand of repocutter to strip boring
commits out of a Subversion dump. For a git dump, look at reposurgeon's
"<<strip_cmd>> --reduce" command.

==== Prune irrelevant branches

Try to throw away branches that are not relevant to your problem.  The
'expunge' operation on repocutter or the '<<delete_cmd>> branch'
command in reposurgeon may be helpful.

This is the attempted simplification least unlikely to make your bug
vanish, so check that carefully after each branch deletion.

=== Know how to spot possible importer bugs

If your target VCS's importer dies during a rebuild, try writing the
repository content to a stream instead and importing the stream by
hand.  If the latter does not fail, the target VCS's importer may be
slightly buggy - but you have a workaround.

(This has been observed under git 2.5.0 with the result of a
'<<unite_cmd>>' operation on two repositories. The cause is unknown,
as git dies suddenly enough to not leave a crash report.)

=== Benchmarking

A fair amount of effort has been expended to keep the run-time
performance of reposurgeon as linear as possible. This is not an easy
state to stay in; it is unfortunately quite simple to accidentally
regress this without noticing.

To that end, there are some fairly simple scripts in the bench
directory of the source distribution that can be used to check for
this type of problem. `repobench` runs reposurgeon multiple times with
a different readlimit each time, recording the run time and memory
allocated at each iteration. Supply arguments specifying the svn dump
file to read and the readlimit values to use like this:

----
./repobench your-dump-file.svn 1000 2000 20000
----

This reads `your-dump-file.svn` 10 times, with the readlimit set first
to 1,000, then 3,000, etc, stepping up until it reaches 20,000.

This produces a `.dat` file which you can use with `repobench -p`, or
`repobench -o` to produce graphs.

For an example, see `link:oops.svg[oops.svg]`. This shows a graph made
using a good revision that had linear performance, several made with
revisions that introduced a regression that made performance quite
non-linear, and the fix. You can easily tell the difference visually.

[[changes]]
=== Incompatible language changes

Reposurgeon scripts are effectively never reused. Thus, incompatible
changes to the command language don't have a high cost in pain to
users, and the maintainers feel free to make them whenever improvement
seems possible.  But just in case, such changes are recorded here.

In versions 4.38 and earlier, the DSL was substantially different The
command-line parser was fragile and ad-hoc, without consistent
application of double-quoting. there was no "view" or "clone" command.
The =E and =X event selectors didn't yet exist.  There was a read --use-uuid
option which has been retired. Negation was not yet supported in
pathsets. The regex, path rename, branch rename, and tag rename
commands used Python-style rather than Go-style match
references. Macro argument formals did not yet have a % prefix. The
split command syntax was incompatibly different.  The "set" and
"clear" commands didn't take a subcommand verb.  There were "logfile"
"readlimit" command doing what "set logfile" and "set readlimit" now
does. The "elapsed", "memory", "when", and "sizeof" subcommands of
"show" used to be separate commands. Blob/tag/reset creation used to
be "{blob|tag|reset} create", not "create {blob|tag|reset}"; similarly
for delete, rename, and move. The command to rename repositories was
just "rename", not "rename repo"; The "delete path" command was named
"expunge".  The subcommands of "list" were separate commands.  There
was a "script" command separate from "do". The "reference list"
command was removed (it duplicated the =N delector); "references
lift" became "stampify"; "timings" became "checkpoint"; "attribution"
bnecomes "attribute". The syntax of the "remove" command was different
and more complex. The "import" command was "incorporate". The
"select" command was "sourcetype".

In versions 4.30 and earlier, the read command had a --branchify
option. This has been retired, you are now expected to do the
equivalent manipulations with repocutter.

The "tip" command present in 4.24 and earlier has been removed,
because the concept it's trying to compute is not clearly defined at
any commit upstream of a fork point. There's no evidence this was
ever used in the wild.

In versions 4.24 and earlier the behavior of the "unite" command
was incompatibly different in that it took master branch from the last
repo in the list, renaming others. Now it leaves master from the first.

The blob command now takes an explicit mark, rather than creating a new
blob :1 and renumbering all others as in 4.23 and earlier.

Filter syntax now uses subcommand verbs rather than the options of 4.23 and
earlier.

In versions 4.23 and earlier, the stats command operated on named
repositories rather than the currently-selected one.

In versions 4.23 and earlier, it was possible to redirect output from
msgin to capture a mailbox of changed entries. This feature has been
removed; msgin now sets =Q bits which can be used to generate the
same report.

In versions 4.23 and earlier, "<<branchify_opt>>" read
option was a separate command that needed to be invoked before the
read and set hidden global context.  There was a related branchmap
command that has been retired because branch rename covers all
its cases in a simpler way.

In versions 4.23 and earlier, the behavior of the branch command was
incompatibly different; it did not require a "heads/" or "tags/"
prefix on its operand and, because of that, could only operate on
heads/ branches and not lightweight tags.

In versions 4.23 and earlier, several commands that now have the form
"object verb selection" had the form "object selection verb".  This
includes branch, tag, reset, and path.

In versions 4.23 and earlier, the syntax of the expunge command
was different; it used "~" instead of "--not" and took multiple
patterns.

The "paths sup" and "paths sub" commands of versions 4.23 and earlier
have been retired and replaced by the enhanced path rename command.

As of 4.23, branch references as used in "branch rename" and elsewhere
now require a namespace prefix - heads/ or tags/.  This was done to
eliminate a number of special cases that were difficult to document
and remember.

In versions before 4.10, the "reduce" and "blob" options of the
"<<strip_cmd>>" command were bare keywords. Also the options of the
"<<ignores_cmd>>" command were bare keywords. There was a command to
set the prompt string that has been retired.

In versions before 4.8, the `expunge` command run on a
repository named "foo" tried to keep deleted fileops in a new
synthetic repository named "foo-expunges".  This feature has been
replaced by the "~" negation operator on expunge selections.

In versions before 4.1, the `<<index_cmd>>` command did not see blobs
by default.

In versions before 4.0, `<<msgin_cmd>>` and `<<msgout_cmd>>` were
named "mailbox_in" and "mailbox_out:"; `--branchify` was
"branchify_map". Previous versions used the Python variant of regular
expressions; some of the more idiosyncratic features of these are not
replicated in the Go implementation.

In versions before 3.23, '```<<prefer_cmd>>```' changed the
repository type as well as the preferred output format.
Since then, do this with '```<<select_cmd>>```'.

In versions before 3.0, the general command syntax put the
command verb first, then the selection set (if any) then modifiers
(VSO).  It has changed to optional selection set first, then command
verb, then modifiers (SVO). The change made parsing simpler, allowed
abolishing some noise keywords, and recapitulates a successful design
pattern in some other Unix tools - notably
sed(1).

In versions before 3.0, path expressions only matched
commits, not commits and the associated blobs as well. The names
of the "a" and "c" flags were different.

In reposurgeon versions before 3.0, the `<<delete_cmd>>` command had
the semantics of `<<squash_cmd>>`; also, the policy flags did not
require a '```--```' prefix. The '```--delete```' flag was named
"obliterate".

In reposurgeon versions before 3.0, `<<read_cmd>>` and `<<write_cmd>>`
optionally took file arguments rather than requiring redirects (and
the `<<write_cmd>>` command never wrote into directories). This was
changed in order to allow these commands to have modifiers. These
modifiers replaced several global options that no longer exist.

In reposurgeon versions before 3.0, the earliest factor in a
`<<unite_cmd>>` command always kept its tag and branch names
unaltered. The new rule for resolving name conflicts, giving priority
to the latest factor, produces more natural behavior when uniting two
repositories end to end; the master branch of the second (later) one
keeps its name.

In reposurgeon versions before 3.0, the `<<tagify_cmd>>`
command expected policies as trailing arguments to alter its behavior. The new
syntax uses similarly named options with leading dashes, that can appear
anywhere after the `<<tagify_cmd>>` command.

In versions before 2.9. the syntax of `<<authors_cmd>>`,
`<<legacy_cmd>>`, `<<list_cmd>>`, and what are now `<<msgin_cmd>>` and
`<<msgout_cmd>>` was different (and `<<legacy_cmd>>` was named
`fossils`). They took plain filename arguments rather than using
redirect < and >.

In versions so old that the changeover point is now lost in the mists
of time, curly brackets (not parens) performed sub-expression grouping.


[[emergency]]
=== Emergency help

If you need emergency help, go to the #reposurgeon IRC on irc.oftc.net.
Be aware, however, that the maintainer is too busy to babysit
difficult repository conversions unless he has explicitly volunteered
for one or someone is paying him to care about it.  For explanation,
see http://esr.ibiblio.org/?p=5087[Your money or your spec].

[[extensions]]
== Stream syntax extensions

The event-stream parser in `reposurgeon` supports some extended
syntax. Exporters designed to work with `reposurgeon` may have a
`--reposurgeon` option that enables emission of extended syntax;
notably, this is true of cvs-fast-export(1).  The remainder of this
section describes these syntax extensions.  The properties they set
are (usually) preserved and re-output when the stream file is written.

The token '```#reposurgeon```' at the start of a comment line in a
fast-import stream signals reposurgeon that the remainder is an
extension command to be interpreted by `reposurgeon`.

One such extension command is implemented: '```sourcetype```', which
behaves identically to the reposurgeon `<<select_cmd>>`
command. An exporter for a version-control system named "frobozz"
could, for example, say

--------
#reposurgeon sourcetype frobozz
--------

Within a commit, a magic comment of the form '```#legacy-id```' declares a
legacy ID from the stream file's source version-control system.

Also accepted is the bzr/brz syntax for setting per-commit
properties. While parsing commit syntax, a line beginning with the
token '```property```' must continue with a whitespace-separated
property-name token. If it is then followed by a newline it is taken
to set that boolean-valued property to true. Otherwise it must be
followed by a numeric token specifying a data length, a space,
following data (which may contain newlines) and a terminating
newline. For example:

--------
commit refs/heads/master
mark :1
committer Eric S. Raymond <esr@thyrsus.com> 1289147634 -0500
data 16
Example commit.

property legacy-id 2 r1
M 644 inline README
--------

Unlike other extensions, bzr/brz properties are only preserved on
stream output if the preferred type is bzr or brz, because any importer other
than those will choke on them.

[[limitations]]
== Limitations and guarantees

Guarantee: In DVCSes that use commit hashes, editing with reposurgeon
never changes the hash of a commit object unless (a) you edit the
commit, or (b) it is a descendant of an edited commit in a VCS that
includes parent hashes in the input of a child object's hash (git and
hg both do this).

Guarantee: reposurgeon only requires main memory proportional to the
size of a repository's metadata history, not its entire content
history. (Exception: the data from inline content is held in memory.)

Guarantee: In the worst case, reposurgeon makes its own copy of every
content blob in the repository's history and thus uses intermediate
disk space approximately equal to the size of a repository's content
history. However, when the repository to be edited is presented as a
stream file, reposurgeon requires no or only very little extra disk
space to represent it; the internal representation of content blobs is
a (seek-offset, length) pair pointing into the stream file.

Guarantee: reposurgeon never modifies the contents of a repository it
reads, nor deletes any repository. The results of surgery are always
expressed in a new repository.

Guarantee: Any line in a fast-import stream that is not a part of a
command reposurgeon parses and understands will be passed through
unaltered.  At present the set of potential passthroughs is known to
include the `progress`, `options`, and `checkpoint` commands as
well as comments led by `#`.

Guarantee: All reposurgeon operations either preserve all repository
state they are not explicitly told to modify or warn you when they
cannot do so.

Guarantee: reposurgeon handles the bzr/brz commit-properties extension,
correctly passing through property items including those with embedded
newlines. (Such properties are also editable in the message-box
format.)

Limitation: Because reposurgeon relies on other programs to generate
and interpret the fast-import command stream, it is subject to bugs in
those programs.

Limitation: bzr/brz suffers from deep confusion over whether its unit of
work is a repository or a floating branch that might have been cloned
from a repo or created from scratch, and might or might not be
destined to be merged to a repo one day. Its exporter only works on
branches, but its importer creates repos.  Thus, a rebuild operation
will produce a subdirectory structure that differs from what you
expect.  Look for your content under the subdirectory '```trunk```'.

Limitation: under git, signed tags are imported verbatim. However, any
operation that modifies any commit upstream of the target of the
tag will invalidate it.

Limitation: Stock git (at least as of version 1.7.3.2) will choke on
property extension commands. Accordingly, reposurgeon
omits them when rebuilding a repo with git type.

Limitation: Converting an hg repo that uses bookmarks (not
branches) to git can lose information; the branch ref that git assigns
to each commit may not be the same as the hg bookmark that was active
when the commit was originally made under hg.  Unfortunately, this is
a real ontological mismatch, not a problem that can be fixed by
cleverness in reposurgeon.

Limitation: Converting an hg repo that uses branches to git can
lose information because git does not store an explicit branch as part
of commit metadata, but colors commits with branch or tag names on the
fly using a specific coloring algorithm, which might not match the explicit
branch assignments to commits in the original hg repo. Reposurgeon
preserves the hg branch information when reading an hg repo, so it is
available from within reposurgeon itself, but there is no way to preserve
it if the repo is written to git.

Limitation: Not all BitKeeper versions have the fast-import and
fast-export commands that reposurgeon requires.
They are present back to the 7.3 open-source version.

Limitation: reposurgeon may misbehave under a filesystem which smashes
case in filenames, or which nominally preserves case but maps names
differing only by case to the same filesystem node (Mac OS X behaves
like this by default).  Problems will arise if any two paths in a repo
differ by case only. To avoid the problem on a Mac, do all your
surgery on an HFS+ file system formatted with case sensitivity
specifically enabled.

Guarantee: As version-control systems add support for the
fast-import format, their repositories will become editable by
reposurgeon.

== Notes on the DSL

The syntax of the reposurgeon DSL evolved ad-hoc as more capabilities
were glued onto the tool. The result has been something of an irregular mess
that is difficult for uses to wrap their heads around even with good
documentation.

Wrestling the DSL into a simpler and more uniform shape is an ongoing
area of development, and not a simple one. There are conflicting
objectives:

. Simplify individual commands with complex syntax.

. Most commands have the form of a simple verb. (V). Some have the
  form verb-subject (VS) or verb-subject-object (VSO). Some have the
  form subject-verb (SV). Some are subject nouns (S).
  We want the simplest and most uniform morphology possible.

. Group commands so the language is easier to remember and document.

. Use subcommand keywords for alternative modes, as opposed to
  policy bits that can be set several at a time.

. Keep commonly-used commands short and simple.

A relatively recent advance at the end of the 4.x release
series was implementing uniform argument-tokenization rules across all
commands (with one motivated exception at "filter shell").  This
replaced differing tokenization rules in different commands.

The state of play...

Most commands are V:

----
{SELECTION} add { "D" {PATH} | "M" {PERM} {MARK} {PATH} | "R" {SOURCE} {TARGET} | "C" {SOURCE} {TARGET} }
SELECTION append [--rstrip] {TEXT}
{SELECTION} assign [--singleton] [NAME]
branchlift SOURCEBRANCH PATHPREFIX [NEWNAME]
{SELECTION} checkout DIRECTORY
checkpoint [MARK-NAME] [>OUTFILE]
choose [REPO-NAME]
[SELECTION] coalesce [--changelog] [--debug] [TIMEFUZZ]
{SELECTION} count [>OUTFILE]
debranch SOURCE-BRANCH [TARGET-BRANCH]
[SELECTION] dedup
define [NAME [TEXT]]
{SELECTION} diff [>OUTFILE]
{SELECTION} divide
do MACRO-NAME [ARG...]
drop [REPO-NAME]
exit [>OUTFILE]
gc [GOGC]
[SELECTION] gitify
[SELECTION] graft [--prune] REPO-NAME
[SELECTION] graph [>OUTFILE]
[SELECTION] hash [--tree] [>OUTFILE]
help [COMMAND]
history
{SELECTION} import [--date=YY-MM-DDTHH:MM:SS|--after|--firewall] [TARBALL...]
[SELECTION] lint [--OPTION...] [>OUTFILE]
log [[+-]LOG-CLASS]...
{SELECTION} merge
[SELECTION] msgin [--create] [<INFILE]
[SELECTION] msgout  [--decode=codec] [--filter=PATTERN] [--blobs]
prefer [VCS-NAME]
SELECTION prepend [--rstrip] {TEXT}
preserve [PATH...]
print [TEXT...] [>OUTFILE]
quit
read [--quiet] [<INFILE | - | DIRECTORY]
rebuild [DIRECTORY]
[SELECTION] remove {INDEX | ["D"|"M"|"R"|"C"|"N"] [PATH]} [to TARGET]
renumber
[SELECTION] reorder [--quiet]
{SELECTION} resolve
select [VCS-NAME]
[SELECTION] setfield FIELD VALUE
{SELECTION} setperm PERM [PATH...]
shell [COMMAND-TEXT]
[SELECTION] split [ --path ] PATH-OR-INDEX
[SELECTION] squash [POLICY-FLAGS...]
[SELECTION] stampify
[SELECTION] strip {--reduce|--blobs|--obscure}
[SELECTION] tagify [ --tagify-merges | --canonicalize | --tipdeletes ]
[SELECTION] transcode ENCODING
unassign NAME
undefine MACRO-NAME
unite [--prune] [REPO-NAME...]
{SELECTION} unmerge
unpreserve [PATH...]
view [directory]
[SELECTION] write [--legacy] [--noincremental] [--callout] [>OUTFILE|-|DIRECTORY]
----

VS:

----
[SELECTION] attribute [ATTR-SELECTION] SUBCOMMAND [ARG...]
clear flag [canonicalize|crlf|compress|echo|experimental|interactive|progress|serial|faketime|quiet]+
clear {logfile|readlimit|desclimit}
[SELECTION] create {repo NAME|blob NAME [<INFILE]|tag NAME|reset NAME}
{SELECTION} delete {commit | {path|tag|branch|reset} [--quiet|--not|--notagify] PATTERN]}
[SELECTION] filter {dedos|shell|regexp|replace} [TEXT-OR-REGEXP]
[SELECTION] list [--decode=codec] [commits|tags|stamps|inspect|index|manifest|paths|names] [PATTERN] [>OUTFILE]
profile {live|start|save|bench} [PORT | SUBJECT [FILENAME]]
set flag [canonicalize|crlf|compress|echo|experimental|interactive|progress|serial|faketime|quiet]+
set {logfile|readlimit|desclimit} VALUE
show {elapsed|memory|sizeof|when TIMESTAMP} [>OUTFILE]
----

VSO:

----
[SELECTION] move {tag|reset} [--not] PATTERN [NEW-NAME]
[SELECTION] remove {INDEX | ["D"|"M"|"R"|"C"|"N"] [PATH]} [to TARGET]
[SELECTION] rename {repo | path PATTERN [--force] | {path|branch|tag|reset} [--not] PATTERN}} NEW-NAME
----

The following are SV. Inverting these to verb-first might be a good
thing. Unfortunately this would make the embedded help for the read
and write verbs a big heterogenous mess.

----
[SELECTION] authors {read <INFILE | write >OUTFILE}
legacy {read [<INFILE] | write [>OUTFILE]}
----

The following are S, and there's no obvious way to
move them to a verb-first form:

----
[SELECTION] changelogs [BASENAME-PATTERN]
ignores [--translate] [--defaults]
[SELECTION] timeoffset {OFFSET}
[SELECTION] timequake [--tick]
version [EXPECT]
----

== Credits

These are in roughly descending magnitude.

Eric S. Raymond <esr@thyrsus.com>::
   Designer and original author.

Julien "_FrnchFrgg_" RIVAUD <frnchfrgg@free.fr>::
   Lots of high-quality code cleanups and speed tuning. Responsible for at
   least half of the massively revamped Subversion dump reader on the 4.0
   releases. Ported the CoW filemaps from Python to Go.

Daniel Brooks <db48x@db48x.net>::
   Date unit testing, improvements for split and expunge
   commands. Assistance on Python to Go port. Go profiling support.
   Several significant reductions in total run time, total
   allocations, and max heap usage.

Greg Hudson <ghudson@MIT.EDU>::
   Contributed copy-on-write filemaps, which both tremendously sped up
   Subversion dumpfile parsing and squashed a nasty bug in the older
   code. While his CoW implementation was eventually replaced with one
   by Julien Rivaud, it busted the project out of a nearly two-year slump.

Eric Sunshine <sunshine@sunshineco.com>::
   Review of seldom-used features, test improvements, bug-fixing.
   Generalized selection expression parser for use-cases other than events.
   Converted selection parser, which evaluated an expression while parsing it,
   to a compile/evaluate paradigm in which a selection expression can be
   compiled once and evaluated many times.
   Added '<<attribute_cmd>>' command.
   Added '<<reorder_cmd>>' command.
   Assisted Python to Go port.

Edward Cree <ec429@cantab.net>::
   Wrote the Hg extractor class and its test.

Ian Bruene <ianbruene@gmail.com>::
   Wrote the kommandant package and the Go port of Python difflib
   in order to support this package.

Chris Lemmons <alficles@gmail.com>::
   Solved some problems with inline blobs, improved interoperability with
   Mercurial, wrote the --prune option for <<graft_cmd>>.

Richard Hansen <rhansen@rhansen.org>::
   Selections as ordered rather than compulsorily sorted sets. The
   generalized <<reparent_cmd>> command.
   Improvements in regression-test infrastructure.

Peter Donis <peterdonis@alum.mit.edu>::
   Python 3 port and Python2/3 interoperability. Historical: none
   of this survived the port to Go.

[appendix]
== The ontological-mismatch problem and its consequences

There are many tools for converting repositories between
version-control systems out there. This appendix explains why
reposurgeon is the best of breed by comparing it to the
competition.

The problems other repository-translation tools have come from
ontological mismatches between their source and target systems -
models of changesets, branching and tagging can differ in complicated
ways. While these gaps can often be bridged by careful analysis, the
techniques for doing so are algorithmically complex, difficult to
test, and have ugly edge cases.

Furthermore, doing a really high-quality translation often requires
human judgment about how to move artifacts - and what to discard.
But most lifting tools are, unlike reposurgeon, designed as
run-it-once batch processors that can only implement simple
and mechanical rules.

Consequently, most repository-translation tools evade the harder
problems. They produce a sort of pidgin rendering that crudely and
partially copies the history from the source system to the target
without fully translating it into native idioms, leaving behind
metadata that would take more effort to move over or leaving it in
the native format for the source system.

But pidgin repository translations are a kind of friction drag on future
development, and are just plain unpleasant to use.  So instead of evading
the hard problems, reposurgeon provides a power assist for a human
to tackle them head-on.

Here are some specific symptoms of evasion that are common enough to
deserve tags for later reference.

LINEAR: One very common form of evasion is only handling linear
histories.

NO_IGNORES: There are many different mechanisms for ignoring files -
.cvsignore, Subversion svn:ignore properties, .gitignore and their
analogues. Many older Subversion repositories still have .cvsignore
files in them as relics of CVS prehistory that weren't translated when
the repository was lifted.  Reposurgeon, on the other hand, knows
these can be changed to .gitignore files and does it.

NO_TAGS: Many repository translators cannot generate annotated tags
(or their non-git equivalents) even when that would be the right
abstraction in the target system.

CONFIGURATION: Another common failing is for repository-translation
tools to require a lot of configuration and ceremony before they can
operate.  Often, for example, tools that translate from Subversion
repositories require you to declare the repository's branch structure
every time even though sensible defaults and a bit of autodetection
could have avoided this.

MIXED_BRANCH: Yet another case usually handled poorly (in translators
that handle branching) is mixed-branch commits. In Subversion it is
possible (though a bad idea) to commit a changeset that modifies
multiple branches at once.  All sufficiently old Subversion
repositories have these, often by accident.  The proper thing to do
is split these up; the usual thing is to assign them to one branch
and leave them omitted from the others.

Version references in commit comments.  It is not uncommon to see a
lot of references that are no longer usable embedded in translated
repositories like fossils in geological strata - file-version numbers
like '1.2' in Subversion repos that had a former life in CVS,
Subversion references like 'r1234' in git repositories, and so forth.
There's no tag for this because tools other than reposurgeon generally
have no support at all for lifting these.

To avoid repetitive text in these descriptions, we use the following
additional bug tags:

ABANDONED: Effectively abandoned by its maintainer. Some tools with
this tag are still nominally maintained but have not been updated
or released in years.

NO_DOCUMENTATION: Poorly (if at all) documented.

!FOO means the tool is known not to have problem FOO.

?FOO means the author has not tried the tool but has strong reason to
suspect the problem is present based on other things known about it.

You should assume that none of these tools do reference-lifting.

=== cvs2svn

https://github.com/mhagger/cvs2svn

Just after the turn of the 21st century, when Subversion was the
new thing in version control, most projects that were using version
control were using CVS, and cvs2svn was about the only migration path.

Early cvs2svn had problems on every level, only some of which have
been fixed by more recent releases. It tended to spew junk commits
into the translated history, and produced strange combinations of
Subversion internal operations that most later translation tools would
cope with only poorly. Sometimes the resulting translations are
actually malformed; more often they contains noisy commits or commit
duplications that made little sense under Subversion and make even
less under the new target system.

As of late 2023, cvs2svn is "is now in maintenance mode and is not
actively being developed".

!LINEAR, ?MIXED_BRANCH, DOCUMENTATION

=== cvs-fast-export

http://www.catb.org/~esr/cvs-fast-export

Formerly named parsecvs. Originally written by Keith Packard to port
the X.org repositories, which it did a good job on.  Now maintained by
me; reposurgeon uses it to read CVS repositories.  It is *extremely*
fast and can thus be productively used even on huge repositories.

cvs-fast-export is under active maintenance.

!ABANDONED, !LINEAR, !NO_IGNORES, !DOCUMENTATION, !CONFIGURATION

=== cvsps

http://www.catb.org/~esr/cvsps

Don't use this.  Just plain don't.  The author maintained version 3.x
until deprecating it in favor of cvs-fast-export due to
fundamental, unfixable problems.  It gets branch topology wrong in
ways that are difficult to detect.

ABANDONED

=== git-svn

git-svn, the Subversion converter in the git distribution, is really
designed to be a two-way live gateway enabling git users to push and
pull commits from a Subversion server.  It operates by creating a
git repository that is effectively a local mirror of the Subversion
history, then performing Subversion client commands to synchronize
the two in a git-like way.

This choice of mission means that git-svn's translation of history
into git uses a compromise between Subversion idioms and git ones that
is more designed to make transactions back to the Subversion server
easy and safe to generate than it is to make full use of the git
capabilities that Subversion doesn't have.  This is pidgin translation
for a reason better than laziness or failure of nerve, but it's still
pidgin.

Worse, git-svn has bugs that severely compromise it for full
translations.  It tends to stumble over common repository
malformations in Subversion, producing history damage that is
significant but evades superficial scrutiny. The author has written about
this problem in detail at http://esr.ibiblio.org/?p=6778[Don't do
svn-to-git repository conversions with git-svn!]

For a straight linear history with no tags or branches, the difference
between git-svn's Subversion-emulating behavior and the way a git
repository would most naturally be structured is minimal.  But for
conformability with Subversion, git-svn cannot (practically speaking)
use git's annotated-tag facility in the local mirror; instead,
Subversion tags have to be represented in the local mirror as git
branches even if they have no changes after the initial branch copy.

Another thing the live-gatewaying use case prevents is
reference-lifting.  Subversion references like "r1234" in commit
comments have to be left as-is to avoid creating pain for users of the
same Subversion remote not going through git-svn.

git-svn was used by both Google Code's exporter and is used in
GitHub's importer web services.  Depending on the latter is not
recommended.

!ABANDONED, MIXED_BRANCH, NO_TAGS, NO_IGNORES.

=== git-svnimport

Formerly part of the git suite; what they had before git-svn, and inferior to
it.  Among other problems, it can only handle Subversion repos with
a "standard" trunk/tags/branches layout. Now deprecated.

MIXED_BRANCH, NO_TAGS, NO_IGNORES, ABANDONED.

=== git-svn-import

https://github.com/stevenharman/git-svn-import

A trivial wrapper around git-svn.  All the reasons not to use git-svn
apply to it as well. As of late 2023 has not seen an update in ten years.

MIXED_BRANCH, NO_TAGS, NO_IGNORES, ABANDONED.

=== svn-fe

svn-fe was a valiant effort to produce a tool that would dump a
Subversion repository history as a git fast-import stream.  It
made it into the git contrib directory at one point, but
seems to have been dropped.

LINEAR, NO_TAGS, NO_IGNORES, ABANDONED.

=== Tailor

https://github.com/lelit/tailor

Tailor aimed to be an any-to-any repository translator.
As of late 2023 the last commit was in 2011.

LINEAR, ?NO_IGNORES, ABANDONED.

=== agito

https://github.com/fragglet/agito

This is a Subversion-to-git tool that was written to handle some
cases that git-svn barfs on (but reposurgeon doesn't - the reposurgeon
test suite contains a case sent by agito's author to check this).
It even handles mixed-branch commits correctly.

!LINEAR, !NO_TAGS, !MIXED_BRANCH, CONFIGURATION, ABANDONED.

If you could not use reposurgeon for some reason, this was one of
the best alternatives. However, as of 2023 it appears to have
been abandoned in 2016.

=== svn2git (jcoglan/nirvdrum version)

https://github.com/nirvdrum/svn2git

A batch-conversion wrapper around git-svn that creates real tag
objects. This is the one written in Ruby.  Has all of git-svn's
problems, alas.

!ABANDONED, !NO_TAGS, NO_IGNORES, ABANDONED.

=== svn2git (Schemenauer version)

http://python.ca/nas/python/svn2git.py

Native Python. More a proof of concept than a production tool.

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION, ABANDONED.

=== svn-all-fast-export (nyblom version)

https://github.com/svn-all-fast-export/svn2git

Written in C++. Says it's also named svn2git and might be genetically
related to Nyblom svn2git.  Was used to convert the KDE subversion
repository to Git. Looks like it might not be terrible, but requires
very elaborate configuration to make Subversion branch structure to
git branch structure. Actively maintained in 2023.

CONFIGURATION, !ABANDONED.

=== svn-fast-export

Formerly at http://c133.org/code/svn-fast-export.c

Written in C.  More a proof of concept than a production tool.

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION, ABANDONED.

=== svn-dump-fast-export

https://github.com/barrbrain/svn-dump-fast-export/

Written in C.  Documentation is so lacking that
there isn't even a README.  However, it's
possible to deduce what isn't there by
reading the code. In 2023 this effectively moved
into the Git tree.

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION, ABANDONED.

===  svn-all-fast-export (thiago version)

http://repo.or.cz/w/svn-all-fast-export.git

May be genetically related to the Nyblom svn2git, but if so they
diverged in 2008. As of 2023,

LINEAR, NO_TAGS, NO_IGNORES, NO_DOCUMENTATION, ABANDONED.

=== sccs2git

There is a script called sccs2git on CPAN which attempts to lift SCCS
collections direct to Git. It is not recommended, as
it is poorly documented and makes no attempt to group commits into
changesets.

NO_DOCUMENTATION, ABANDONED.

=== SubGit

http://www.subgit.com/

Nearly unique for this category of software in being closed-source.
Beyond an evaluation period, users have to register, possibly for a cost
(it's supposed to be free-of-charge for certain uses: open source
projects, education, and "startups" -- history with BitKeeper shows
that such exemptions should probably not be trusted).

The intended outcome of this program is to provide a server with
support for both Subversion and Git users to interact at once.  This
may be of little value overall, as new developers are frequently
unfamiliar with Subversion (and old ones forget the usage patterns!),
fundamental differences in design of the two VCSes interfering with
the quality of both views, and increased confusion with preferred
modes of contribution arise.

The quality of SubGit's conversion is rather poor. It fails to
properly translate at least half of the reposurgeon *.svn regression
tests, even some of the simpler ones - although trickier cases such as
agito.svn it does translate correctly.  Large real-world Subversion repos
will exhibit multiple issues that SubGit may, silently or otherwise,
trip over.

This program will forever contain compromises for the same reasons
git-svn does.  The non-open source nature leaves little hope of having
such issues repaired by skilled community members.

Atlassian's BitBucket service relies on this for Subversion-to-Git
migration.  Depending on this service is not recommended.

!MIXED_BRANCH, !LINEAR, CONFIGURATION, DOCUMENTATION

[appendix]
== A tour of the codebase

Reposurgeon is intended to be hackable to add support for
special-purpose or custom operations, though it's even better if you
can come up with a new surgical primitive general enough to ship with
the stock version.  For either case, here's a guide to the
architecture.

=== inner.go

The core classes in inner.go support deserializing and reserializing
import streams.  In between these two operations the repo state lives
in a fairly simple object, Repository. The main part of Repository is
just a list of objects implementing the Event interface - Commits,
Blobs, Tags, Resets, and Passthroughs.  These are straightforward
representations of the command types in a Git import stream, with
Passthrough as a way of losslessly conveying lines the parser does not
recognize.

----------------------------------------------------------------------
 +-------------+    +---------+    +-------------+
 | Deserialize |--->| Operate |--->| Reserialize |
 +-------------+    +---------+    +-------------+
----------------------------------------------------------------------

The general theory of reposurgeon is: you deserialize, you do stuff
to the event list that preserves correctness invariants, you
reserialize.  The "do stuff" is mostly not in the core classes, but
there is one major exception.  The primitive to delete a commit and
squash its fileops forwards or backwards is seriously intertwined
with the core classes and actually makes up almost 50% of Repository
by line count.

The rest of the surgical code lives outside the core classes. Most
of it lives in the Reposurgeon class (the command interpreter) or
the RepositoryList class (which encapsulated by-name access to a list
of repositories and also hosts surgical operations involving
multiple repositories). A few bits, like the repository reader and
builder, have enough logic that's independent of these
classes to be factored out of it.

In designing new commands for the interpreter, try hard to keep them
orthogonal to the selection-set code. As often as possible, commands
should all have a similar form with a (single) selection set argument.

VCS is not a core class.  The code for manipulating actual repos is bolted
on the the ends of the pipeline, like this:

----------------------------------------------------------------------
 +--------+    +-------------+    +---------+    +-----------+    +--------+
 | Import |--->| Deserialize |--->| Operate |--->| Serialize |--->| Export |
 +--------+    +-------------+ A  +---------+    +-----------+    +--------+
      +-----------+            |
      | Extractor |------------+
      +-----------+
----------------------------------------------------------------------

The Import and Export boxes call methods in VCS.

=== extractor.go

Extractor classes build the deserialized internal representation directly.
Each extractor class is a set of VCS-specific methods to be used by the
RepoStreamer driver class.  Key detail: when a repository is recognized by
an extractor it sets the repository type to point to the corresponding
VCS instance.

=== reposurgeon.go

All code that knows about the DSL syntax should live in reposurgeon.go
along with the program main and the functions for reporting errors,
logging, handling signals and aborts, etc.

=== svnread.go

This is the reader for Subversion dumpfiles. It is the only exception
to the rule that reads support for version control systems is
implemented by front ends that read them and emit a fast-import
stream.

The reason it's an exception is that Subversion has its own
serialization format, and the total complexity of embedding
support for those streams was estimated to be lower than that
if writing a a completely separate front end.

=== Style notes

The code was translated from Python. It retains, for internal
documentation purposes, the Python convention of using leading
underscores on field names to flag fields that should never be
referenced outside a method of the associated struct.

The capitalization of other fieldnames looks inconsistent because
the code tries to retain the lowercase Python names and
compartmentalize as much as possible to be visible only within the
declaring package.  Some fields are capitalized for backwards
compatibility with the setfield command in the Python
implementation, others (like some members of FileOp) because
there's an internal requirement that they be settable by the Go
reflection primitives.

[appendix]
== Adding support for more version-control systems

The best way way to add support for a version-control system not
already on the list is to write a pair of foo-fast-export and
foo-fast-import utilities (separate from reposurgeon) that generate
and consume git fast-import streams. When this is achievable, it
enables full read/write support for repositories of that type. In this
case the supporting changes in reposurgeon will be trivial, just a
pair of table entries.

The next best route is to write a FooExtractor class in reposurgeon
itself. This is less good because (a) it provides only read-side
support, and (b) it adds complexity to reposurgeon.  There's also a
filter derived from testing requirements; a command-line client of
your FooVCS must be freely available running under Unix in order for the
reposurgeon maintainers to run tests on it.  We are not willing to
ship features we can't test.

Finally, if your VCS supports a native serialization format that it
can use as a dump/restore for live repositories, and has or both of a
pair of foo-dump/foo-load utilities analogous to git-fast-export and
git-fast-import, it may be possible to support your FooVCS through
that format. Subversion's svnadmin dump/load commands fit this
pattern.

In this case it is still best to try to write filters that
interconvert between the native serialization and git-fast-import
streams, separately from reposurgeon. This makes the testing problem
more tractable, and means that reposurgeon itself needs only a couple
of additional table entries calling simple pipelines.

As a last resort, the reposurgeon maintainers may consider adding
support for reading and writing a native serialization format
to reposurgeon itself.  So far this has only been done once, for
Subversion, and there is an important precondition; the serialization
format must have complete public documentation.

Be aware that proprietary VCSes in general are likely to cause us
serious testing problems and we are reluctant to try to support
them. If a maintainer has to pay money to have binaries he or she can
run tests with, you will have to pay a maintainer money to make that
happen.

It's also basically a crash landing if your FooVCS can only be
accessed through a GUI, or its clients only run on Windows, or
it has a CLI that is not capable enough to support an extractor
class. We know of cases where proprietary VCS vendors have
deliberately crippled their export and CLI features in order to
lock customers in; that is no fun to deal with, so you'll
have to pay somebody money.

[appendix]
== Reposurgeon success stories

Reposurgeon has been used for successful conversion on projects
including but not limited to the following.  These are in rough
chronological order.

Hercules (IBM mainframe emulator)::
	 The author did this one, Subversion to hg. About ten years of history
	 at the time, not too horribly messy.

NUT (Network UPS Tools)::
	The author did this one, Subversion to git. The trial by fire - it was
	when the Subversion dump analyzer got built. Very large old
	repository with lots of pathologies (there was a CVS stratum).

Battle For Wesnoth::
	The author did this one, Subversion to git. Very large repo, moderately complex.

Roundup (issue tracker)::
	The author did this one, Subversion to git (they later switched
	to hg). Moderate-sized Subversion repo with some very strange
	malformations.

robotfindskitten::
	The author did this one, CVS to git.  Simple history, pretty easy.

Blender::
	Two guys at Blender did this one with help from the author, Subversion
	to git. Huge repository with a lot of nasty pathologies.  The
	tool needed some serious optimization and feature upgrades
	to handle it.

groff::
	The author did this one, CVS to git.  Rather easy as the project
	history was almost linear and, though very old, not huge.

Nethack::
	CVS to git. This conversion has not yet been publicly released
	at time of writing (late October 2014) for complicated
	political reasons.

Emacs::
	A record three layers, Bazaar over CVS over RCS.
	Malformations not too bad except for some unique challenges
	created by the RCS-to-CVS conversion, but the sheer size of
	the history and number of layers makes it the most complex
	conversion yet. Converted in 2011.

ntp::
	The author did this, BitKeeper to git using a derivative of Tridge's SourcePuller
	as a front end, done in early 2015.  Nothing especially taxing
	about the reposurgeon side of things, the magic was all in the
	front end.

pdfrw, playtag, pyeda, rson::
       Four small Subversion projects by Patrick Maupin, converted in
       two hours' work in May 2015.  No significant difficulties.
       These mainly served to demonstrate that the standard conversion
       workflow in conversion.mk is fast and effective for a wide
       range of projects.

mh-e::
	The Emacs interface for MH.  Converted by Bill Wohler in late 2015.
        He reports that the standard conversion workflow worked fine.

GNUPLOT::
	CVS to git, 30 years of history with some early releases recovered
	from tarballs. Converted by the author in late 2017. Somewhat messy due to
	vendor-branch issues.

GCC::
	SVN to git, with ancient strata of CVS and RCS. 280K commits of
	history back to 1987, dwarfing Emacs. Converted by myself and
	two core GCC developers. The 4.0 release came out of this.
	Final cutover was on Jan 12th 2020.

Here are some other some other field reports on successful uses:

https://blog.runtux.com/posts/2014/04/18/233/[Migrating to GIT with Reposurgeon (2014)]

https://blogs.ed.ac.uk/timc/2017/11/24/migrating-from-svn-to-git-while-splitting-repository/[Migrating from SVN to Git while Splitting Repository (2017)]

[appendix]
== Development History

Links to notable blog posts during the development of reposurgeon.
Trivial release announcements have been omitted.

http://esr.ibiblio.org/?p=2718[Announcing reposurgeon -- a tool for the good new days] (2010-11-01)

// Recorded, not visible because trivial
//http://esr.ibiblio.org/?p=3832[Return of the reposturgeon!] (2011-10-19)

// Recorded, not visible because trivial
//http://esr.ibiblio.org/?p=3844[Bride of the reposturgeon!] (2011-10-25)

http://esr.ibiblio.org/?p=3859[Repositories in Translation] (2011-10-29)

http://esr.ibiblio.org/?p=3897[Revenge of the reposturgeon!] (2011-11-02)

// Recorded, not visible because trivial
//http://esr.ibiblio.org/?p=3905[Son of the reposturgeon!] (2011-11-03)

http://esr.ibiblio.org/?p=2727[Lessons learned from reposurgeon] (2010-11-09)

http://esr.ibiblio.org/?p=3839[Looking for reposurgeon test cases] (2011-10-23)

http://esr.ibiblio.org/?p=4004[Coming soon: reposurgeon does Subversion] (2011-12-12)

// Recorded, not visible because trivial
//http://esr.ibiblio.org/?p=3912[Night of the living reposturgeon!] (2011-11-07)

// Recorded, not visible because trivial
//http://esr.ibiblio.org/?p=3920[Reposturgeon from the Black Lagoon!] ( 2011-11-10)

// Recorded, not visible because trivial
//http://esr.ibiblio.org/?p=3927[Attack of the 50-foot reposturgeon] (2011-11-14)

// Recorded for completeness but not visible as SVN write support was later dropped
//http://esr.ibiblio.org/?p=4721[Beware! The Reposturgeon!] (2012-12-02)

http://esr.ibiblio.org/?p=4691[reposurgeon 2.0 announcement – the full-orchestra version] (2012-11-04)

http://esr.ibiblio.org/?p=4735[The Reposturgeon That Ate Sheboygan!] (2012-12-16)

http://esr.ibiblio.org/?p=4738[Reposturgeon Attacks Tokyo!] (2012-12-20)

http://esr.ibiblio.org/?p=4748[Reposurgeon Killed The Radio Stars!] (2013-01-08)

http://esr.ibiblio.org/?p=4071[Another bite of the reposturgeon] (2012-01-23)

http://esr.ibiblio.org/?p=4810[The Reposturgeon from Beyond Space!] (2013-02-08)

http://esr.ibiblio.org/?p=4854[Mode of the Reposturgeon!] (2013-02-26)

http://esr.ibiblio.org/?p=5123[Reposurgeon Battles All Monsters!] (2013-12-01)

http://esr.ibiblio.org/?p=5087[Your money or your spec] (2013-10-29)

http://esr.ibiblio.org/?p=5135[Heads up: the reposturgeon is mutating!] (2013-12-05)

http://esr.ibiblio.org/?p=5634[Ugliest...repository...conversion...ever] (2014-03-29)

http://esr.ibiblio.org/?p=5185[Reposturgeon and Santa Claus Against The Martians!] (2014-01-05)

http://esr.ibiblio.org/?p=6524[Emacs git conversion is done] (2014-11-12)

http://esr.ibiblio.org/?p=6778[Don’t do svn-to-git repository conversions with git-svn!] (2015-05-28)

http://esr.ibiblio.org/?p=6792[How to spot a high-quality repository conversion] (2015-06-23)

http://esr.ibiblio.org/?p=6971[Cometary Contributors] (2016-01-10)

http://esr.ibiblio.org/?p=7032[Automatons, judgment amplifiers, and DSLs] (2016-02-17)

http://esr.ibiblio.org/?p=7792[You’re gonna need a bigger Beast] (2017-12-16)

http://esr.ibiblio.org/?p=7403[Reposturgeon recruits the BitKeeper!] (2017-03-06) 

http://esr.ibiblio.org/?p=7804[C, Python, Go, and the Generalized Greenspun Law] (2017-12-18)

http://esr.ibiblio.org/?p=7959[Flight of the reposturgeon!] (2018-04-29)

http://esr.ibiblio.org/?p=8161[Reposurgeon’s Excellent Journey and the Waning of Python] (2018-10-08)

http://esr.ibiblio.org/?p=8543[30 Days in the Hole] (2020-01-24)

http://esr.ibiblio.org/?p=8551[Missing documentation and the reproduction problem] (2020-01-26)

http://esr.ibiblio.org/?p=8607[Reposurgeon defeats all monsters!] (2020-02-21)

http://esr.ibiblio.org/?p=8694[Two graceful finishes] (2020-05-13)

// end
