Prev: LICENSE questions
Next: deskutils/alexandria: maintainer timeout more than 1 month
From: "Julian H. Stacey" on 28 Jul 2010 11:11
Marcin Wisnicki wrote:
> On Wed, 28 Jul 2010 15:43:43 +0200, Julian H. Stacey wrote:
>
> >> Mirroring must be atomic (mirror to temporary directory then rename).
> >
> > That could double disc requirement, & reduce mirror sites willing to
> > provide space ?
>
> Indeed it would. But there is no other way to solve it.
>
> Alternatively it could be prevented with no disk cost by putting
> "MIRROR-IN-PROGRESS" file and making all package utilities check for its
> existence and fail with explanation to try again later.

Perhaps a less visible &/or top listing semaphore of ./.MIRROR-IN-PROGRESS ?
Content could be empty, or mirror master could generate into it, eg:
Version: Number of format of .MIRROR-IN-PROGRESS\n
Master-URL: Of master\n
Protocol: Of mirror\n
Date: (GMT) start of mirror\n
Contact: If mirror fails\n
Text: Any other text.\n
EOR: End of record\n

Various FreeBSD etc stuff gets mirrored as nested directories, so
one might want to to be able to check up to N steps up the parent
tree structure for lock(s).

One might also want an optional force- fetch- regardless flag.

Awkward conditions to allow for:
What if mirror is not mirroring when you start your fetch,
but is when you finish ?
What if mirror is not mirroring when you start your fetch,
then mirror starts & finishes a mirror before you finish your fetch ?

One might want to generalise a mechanism that could be added to
multiple tools, eg ports/net/ mirror/ rdist6/ rsync/ ?...

http://wiki.freebsd.org/FreeBSDMirrors by EdwinGroothuis (edwin@)
mentions orphan/ stale mirrors. Perhaps some are even cascaded mirrors ?

I don't see anything in:
http://www.freebsd.org/projects/ideas/ideas.html
"The FreeBSD list of projects and ideas for volunteers"
Maybe someone might add something ?

See also:
mirror-admin(a)freebsd.org
http://wiki.freebsd.org/IvanVoras/pkg_patchhttp://wiki.freebsd.org/IvanVoras/pkg_patch

Cheers,
Julian
--
Julian Stacey: BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com
Mail plain text. Not HTML, Not quoted-printable, Not Base64.
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Dominic Fandrey on 28 Jul 2010 11:39
On 28/07/2010 15:15, Marcin Wisnicki wrote:
> On Tue, 27 Jul 2010 21:03:21 -0700, perryh wrote:
>
>> Marcin Wisnicki <mwisnicki+freebsd(a)gmail.com> wrote:
>>> At this very moment, french package mirror has INDEX newer than in
>>> other mirrors:
>>>
>> ...
>>>
>>> yet it does not have those packages.
>>>
>>> How could something like this happen ?
>>
>> By being examined while a resync was in process: evidently the new INDEX
>> file had been transferred but that package file (and likely others) were
>> still in transit or perhaps not even started yet. Mirroring is not an
>> instantaneous process.
>
> Yeah that was it, but it is really, really bad.
> Mirroring must be atomic (mirror to temporary directory then rename).
> Otherwise there is a large window of time every couple of days when upgrading
> packages will at best fail or leave you with broken system.
> I did binary upgrade with pkg_upgrade yesterday and half of my system was linked
> against wrong libintl version :(

The next version of pkg_upgrade will check every downloaded package
against the master server after completing the download.

I expect to release it at the end of September.

Regards

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Marcin Wisnicki on 28 Jul 2010 11:58
On Wed, 28 Jul 2010 17:39:17 +0200, Dominic Fandrey wrote:

> On 28/07/2010 15:15, Marcin Wisnicki wrote:
>> On Tue, 27 Jul 2010 21:03:21 -0700, perryh wrote:
>>
>>> Marcin Wisnicki <mwisnicki+freebsd(a)gmail.com> wrote:
>>>> At this very moment, french package mirror has INDEX newer than in
>>>> other mirrors:
>>>>
>>> ...
>>>>
>>>> yet it does not have those packages.
>>>>
>>>> How could something like this happen ?
>>>
>>> By being examined while a resync was in process: evidently the new
>>> INDEX file had been transferred but that package file (and likely
>>> others) were still in transit or perhaps not even started yet.
>>> Mirroring is not an instantaneous process.
>>
>> Yeah that was it, but it is really, really bad. Mirroring must be
>> atomic (mirror to temporary directory then rename). Otherwise there is
>> a large window of time every couple of days when upgrading packages
>> will at best fail or leave you with broken system. I did binary upgrade
>> with pkg_upgrade yesterday and half of my system was linked against
>> wrong libintl version :(
>
> The next version of pkg_upgrade will check every downloaded package
> against the master server after completing the download.
>

Great, I need this so much. Currently to really upgrade packages I must
do something like:

rm -rf /usr/ports/packages
pkg_upgrade -af

to make sure rebuilt packages are refetched.
But it's obviously suboptimal.

I think you could also detect inconsistent mirror by comparing
modification time of package against mtime of INDEX.
If pkg is newer than INDEX then it's a sign of incomplete sync.

> I expect to release it at the end of September.
>
> Regards


_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Marcin Wisnicki on 28 Jul 2010 13:19
On Wed, 28 Jul 2010 17:11:25 +0200, Julian H. Stacey wrote:

> Marcin Wisnicki wrote:
>>
>> Alternatively it could be prevented with no disk cost by putting
>> "MIRROR-IN-PROGRESS" file and making all package utilities check for
>> its existence and fail with explanation to try again later.
>
> Perhaps a less visible &/or top listing semaphore of
> ./.MIRROR-IN-PROGRESS ? Content could be empty, or mirror master could
> generate into it, eg:
> Version: Number of format of .MIRROR-IN-PROGRESS\n Master-URL: Of
> master\n
> Protocol: Of mirror\n
> Date: (GMT) start of mirror\n
> Contact: If mirror fails\n
> Text: Any other text.\n
> EOR: End of record\n
>
> Various FreeBSD etc stuff gets mirrored as nested directories, so one
> might want to to be able to check up to N steps up the parent tree
> structure for lock(s).
>

Unfortunately no scheme involving global lock files will really work.
I've read http://wiki.freebsd.org/FreeBSDMirrors that you linked and it mentions
TIMESTAMP files that it uses to monitor mirrors quality.

You just need to visit $random[1] freebsd mirror and look at that file
then check packages directory. You will find that it can have latest
timestamp yet many months old files.
In other words, mirrors cannot be trusted.

For now however I've found and suggested in sibling post a workaround
that should mostly work assuming INDEX file gets mirrored before packages
(and it looks like it is).

[1] random = ftp.pl.freebsd.org, but it's not the only one

> One might also want an optional force- fetch- regardless flag.
>
> Awkward conditions to allow for:
> What if mirror is not mirroring when you start your fetch, but is when
> you finish ?
> What if mirror is not mirroring when you start your fetch, then mirror
> starts & finishes a mirror before you finish your fetch ?
>

Solving any of this, in a way that does not require intervention of user if it
happens, requires that file content at given url is immutable (or doesn't exist).
There is simply no other 100% reliable way.

You can however do some sanity checks and fail early if failing is acceptable.
Some ideas:
1. if (mtime(pkg) > mtime(INDEX)) fail("Mirror sync in progress");
2. Fetch timestamps of all requested packages in one go before fetching contents;
do it again and compare to be sure nothing changed;
fetch actual packages and fail if mtime/size does not match.
3. Do 1 or 2 but fetch timestamps from well known source or multiple sources.
You may also find most up-to-date mirror this way.

> One might want to generalise a mechanism that could be added to multiple
> tools, eg ports/net/ mirror/ rdist6/ rsync/ ?...
>

_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Dominic Fandrey on 28 Jul 2010 14:02
On 28/07/2010 17:58, Marcin Wisnicki wrote:
> On Wed, 28 Jul 2010 17:39:17 +0200, Dominic Fandrey wrote:
>
>> On 28/07/2010 15:15, Marcin Wisnicki wrote:
>>> On Tue, 27 Jul 2010 21:03:21 -0700, perryh wrote:
>>>
>>>> Marcin Wisnicki <mwisnicki+freebsd(a)gmail.com> wrote:
>>>>> At this very moment, french package mirror has INDEX newer than in
>>>>> other mirrors:
>>>>>
>>>> ...
>>>>>
>>>>> yet it does not have those packages.
>>>>>
>>>>> How could something like this happen ?
>>>>
>>>> By being examined while a resync was in process: evidently the new
>>>> INDEX file had been transferred but that package file (and likely
>>>> others) were still in transit or perhaps not even started yet.
>>>> Mirroring is not an instantaneous process.
>>>
>>> Yeah that was it, but it is really, really bad. Mirroring must be
>>> atomic (mirror to temporary directory then rename). Otherwise there is
>>> a large window of time every couple of days when upgrading packages
>>> will at best fail or leave you with broken system. I did binary upgrade
>>> with pkg_upgrade yesterday and half of my system was linked against
>>> wrong libintl version :(
>>
>> The next version of pkg_upgrade will check every downloaded package
>> against the master server after completing the download.
>>
>
> Great, I need this so much. Currently to really upgrade packages I must
> do something like:
>
> rm -rf /usr/ports/packages
> pkg_upgrade -af
>
> to make sure rebuilt packages are refetched.
> But it's obviously suboptimal.
>
> I think you could also detect inconsistent mirror by comparing
> modification time of package against mtime of INDEX.
> If pkg is newer than INDEX then it's a sign of incomplete sync.

True, but if the check does not fail it still doesn't mean that
the package is valid. Checking against the master (it will check
size and time) is the safest method, I think

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: LICENSE questions
Next: deskutils/alexandria: maintainer timeout more than 1 month