posix threading plans

[Joseph S. Myers]

On Sun, 6 May 2007, Steven J. Hill wrote:

> > The ARM NPTL work was based on trunk at that time (and subsequently merged 
> > from newer trunk) precisely because the incomplete and undocumented merges 
> > made it infeasible to work based on the NPTL branch without getting 
> > regressions relative to trunk.
> > 
> No, when I began the NPTL work there was no one from Code Sourcery that
> even hinted another architecture was being worked on. There was no

We made an announcement when we started the work 
<http://www.uclibc.org/lists/uclibc/2006-May/015441.html>.  We studied the 
state of the branch available at the time, going through the tens of 
thousands of lines of diffs between trunk and branch line by line.  We 
established that:

* Merges had been incomplete and undocumented so there was no trunk 
revision we could compare against and get the NPTL changes and only the 
NPTL changes.

* The branch was in an extremely incomplete state and large parts of the 
port had to be reimplemented because they were not publically available on 
the branch within the required timescale.  We duly reimplemented these, so 
that when we first posted the port is was complete and usable.

I believe we also asked you for any documentation you might have of the 
exact revisions and parts of revisions that had been merged or had yet to 
be merged to the branch, so the limitations of the merges could be 
repaired, but such documentation was not available, so the line-by-line 
comparison was necessary in order to update to then-current trunk.

> initial collaboration and you guys went your own way. The SuperH people

Our work was based on all the sources available at the time from the 
branch, but it required ARM changes that had been made on the trunk since 
the branch had been made, and only parts of those had been inconsistently 
merged to the branch.  Because of the state of the merges to the branch 
and the requirement to be based on more recent sources than those on which 
the branch had been based, we went through the tens of thousands of lines 
of diffs by hand, identified those that represented genuine logical 
changes and updated them to be based on current trunk, thereby creating a 
three-way merge of:

* Current trunk (complete current trunk, not an undocumented mixture of 
different trunk versions):

* NPTL branch (that is, the changes available to the public on the branch 
at the time the development was required to be done).

* The ARM NPTL changes.

In fact the port was prepared twice: initially we based it on the branch, 
reimplementing all parts that had not been committed to the branch at that 
time and merging sufficient of the unmerged trunk changes to get something 
approximately working.  But the requirement was for a port that did not 
regress relative to a more recent version of trunk than that on which the 
branch was based, so we went through all the diffs to identify those 
representing genuine NPTL changes, and merged those to a sufficiently 
recent version of trunk.  The port has been updated several times since 
then.  Each update posted has been based on an identified revision of 
trunk; there are no changes from that revision of trunk unmerged, and no 
changes from later revisions merged, so the diffs represent solely what is 
required to implement that port against that revision of trunk, without 
any extraneous reversions of unmerged trunk changes.

> actually did their work against the NPTL branch and kept me abreast of
> the work and made patches and development against the branch.

Our work was based on the state of the branch at the time, but we were 
required to merge it with current trunk and the state of the branch meant 
the only way of doing this was going through all the diffs and removing 
those that were reversions of unmerged patches.  Since then, we have 
maintained the patch on a basis of current trunk.

If the NPTL branch had been exactly merged from a documented revision of 
trunk, so that every change from that revision of trunk were properly on 
the branch and the only diffs between that revision of trunk and the 
branch were the NPTL ones, and all trunk merges had followed that 
principle, it would have been more feasible to base work on the branch.  
But the branch was not in that state and is not in that state, and 
limitations of SVN make it impracticable to put a branch in that state if 
it gets out of it without as much effort as creating a new branch and 
applying patches by hand.

What we need now is a multi-way merge of:

* Current trunk;

* NPTL branch, as it is now;

* All the various ports.

(Our merge is one of current trunk, as of an identified revision; NPTL 
branch, as it was when we started the work, and the ARM port.)

I believe that to create the required merge, while guaranteeing not to 
accidentally revert any trunk change in the process, requires a new 
branch, where patches are applied one by one after line-by-line audits to 
make sure they do not contain any accidental reversions of past trunk 
changes.  Each commit should be a logical unit, properly explained and 
posted to the list; it may be combined from many separate changes to the 
original branches.

There is no automated way of extracting such clean diffs from SVN, because 
diffing the NPTL branch against *any* revision of trunk will produce a 
mixture of NPTL branch changes, reversions of trunk changes and (depending 
on the date) changes already on trunk since that particular revision.  
Such diffs can only be a starting point for line-by-line editing and 
checking - although new files added on the branch, for example, could 
probably be added at one go without such detailed checks.

It is also necessary to carry out a technical comparison of the different 
approaches to the parts of the NPTL port that had not been publically 
implemented when we began the ARM work: we did one implementation to 
complete the ARM port within the timescale required, and you did another 
one to complete the MIPS port (or committed one that had been done 
previously but not made available before), and for each area where there 
are different approaches they both need to be considered on their merits.  
Some of these have been previously mentioned on the lists.  For example, 
<http://www.uclibc.org/lists/uclibc/2006-August/016237.html>.

-- 
Joseph S. Myers
joseph at codesourcery.com