Prev: LICENSE questions
Next: deskutils/alexandria: maintainer timeout more than 1 month
From: Marcin Wisnicki on 28 Jul 2010 17:36
On Wed, 28 Jul 2010 20:02:40 +0200, Dominic Fandrey wrote:

> On 28/07/2010 17:58, Marcin Wisnicki wrote:
>>
>> I think you could also detect inconsistent mirror by comparing
>> modification time of package against mtime of INDEX. If pkg is newer
>> than INDEX then it's a sign of incomplete sync.
>
> True, but if the check does not fail it still doesn't mean that the
> package is valid. Checking against the master (it will check size and
> time) is the safest method, I think

What do you mean by master ? If you think about ftp-master then it's
password protected. AFAIK there is no publicly available ftp site
that is an authoritative source, ftp.freebsd.org is synced just like
any other mirror.
You can see in my initial post that it was even less up-to-date than
ftp.fr.freebsd.org.

_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: "Andrew W. Nosenko" on 28 Jul 2010 17:24
On Wed, Jul 28, 2010 at 18:39, Dominic Fandrey <kamikaze(a)bsdforen.de> wrote:
> On 28/07/2010 15:15, Marcin Wisnicki wrote:
>> On Tue, 27 Jul 2010 21:03:21 -0700, perryh wrote:
>>
>>> Marcin Wisnicki <mwisnicki+freebsd(a)gmail.com> wrote:
>>>> At this very moment, french package mirror has INDEX newer than in
>>>> other mirrors:
>>>>
>>> ...
>>>>
>>>> yet it does not have those packages.
>>>>
>>>> How could something like this happen ?
>>>
>>> By being examined while a resync was in process: evidently the new INDEX
>>> file had been transferred but that package file (and likely others) were
>>> still in transit or perhaps not even started yet. Mirroring is not an
>>> instantaneous process.
>>
>> Yeah that was it, but it is really, really bad.
>> Mirroring must be atomic (mirror to temporary directory then rename).
>> Otherwise there is a large window of time every couple of days when upgrading
>> packages will at best fail or leave you with broken system.
>> I did binary upgrade with pkg_upgrade yesterday and half of my system was linked
>> against wrong libintl version :(
>
> The next version of pkg_upgrade will check every downloaded package
> against the master server after completing the download.

Excuse me? The ports check downloaded source tarball against SHA
checksum. Just for nay case like downloading error or malicious
inject. Did you try to say that binary package have no such
safeguard?

> I expect to release it at the end of September.

--
Andrew W. Nosenko <andrew.w.nosenko(a)gmail.com>
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Dominic Fandrey on 28 Jul 2010 18:18
On 28/07/2010 23:24, Andrew W. Nosenko wrote:
> On Wed, Jul 28, 2010 at 18:39, Dominic Fandrey <kamikaze(a)bsdforen.de> wrote:
>> On 28/07/2010 15:15, Marcin Wisnicki wrote:
>>> On Tue, 27 Jul 2010 21:03:21 -0700, perryh wrote:
>>>
>>>> Marcin Wisnicki <mwisnicki+freebsd(a)gmail.com> wrote:
>>>>> At this very moment, french package mirror has INDEX newer than in
>>>>> other mirrors:
>>>>>
>>>> ...
>>>>>
>>>>> yet it does not have those packages.
>>>>>
>>>>> How could something like this happen ?
>>>>
>>>> By being examined while a resync was in process: evidently the new INDEX
>>>> file had been transferred but that package file (and likely others) were
>>>> still in transit or perhaps not even started yet. Mirroring is not an
>>>> instantaneous process.
>>>
>>> Yeah that was it, but it is really, really bad.
>>> Mirroring must be atomic (mirror to temporary directory then rename).
>>> Otherwise there is a large window of time every couple of days when upgrading
>>> packages will at best fail or leave you with broken system.
>>> I did binary upgrade with pkg_upgrade yesterday and half of my system was linked
>>> against wrong libintl version :(
>>
>> The next version of pkg_upgrade will check every downloaded package
>> against the master server after completing the download.
>
> Excuse me? The ports check downloaded source tarball against SHA
> checksum. Just for nay case like downloading error or malicious
> inject. Did you try to say that binary package have no such
> safeguard?

Exactly. The INDEX does not contain such information. The thing
is to do that, the pointyhat INDEX format would have to differ
from the ports INDEX format.

A possiblity of course, but also a source of trouble if the INDEX
format of the ports should ever change, something I desire:
http://www.freebsd.org/cgi/query-pr.cgi?pr=148783

Another solution would be to add an empty column that pointyhat
can fill in.

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Dominic Fandrey on 28 Jul 2010 18:23
On 28/07/2010 23:36, Marcin Wisnicki wrote:
> On Wed, 28 Jul 2010 20:02:40 +0200, Dominic Fandrey wrote:
>
>> On 28/07/2010 17:58, Marcin Wisnicki wrote:
>>>
>>> I think you could also detect inconsistent mirror by comparing
>>> modification time of package against mtime of INDEX. If pkg is newer
>>> than INDEX then it's a sign of incomplete sync.
>>
>> True, but if the check does not fail it still doesn't mean that the
>> package is valid. Checking against the master (it will check size and
>> time) is the safest method, I think
>
> What do you mean by master ? If you think about ftp-master then it's
> password protected. AFAIK there is no publicly available ftp site
> that is an authoritative source, ftp.freebsd.org is synced just like
> any other mirror.
> You can see in my initial post that it was even less up-to-date than
> ftp.fr.freebsd.org.

Well, I never had a download of an INDEXED package fail from one
of the unnumbered servers, so there ought to be something different,
because the remaining mirrors normally only are able to provide
~20% of them.

Any way, in pkg_upgrade terms master is whichever server is chosen
to provide the index. Everything else downloaded will be checked to
be consistent with that one.

It's not about getting the latest packages, but about getting a
consistent set.

--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Marcin Wisnicki on 28 Jul 2010 19:29
On Thu, 29 Jul 2010 00:18:01 +0200, Dominic Fandrey wrote:

> On 28/07/2010 23:24, Andrew W. Nosenko wrote:
>>
>> Excuse me? The ports check downloaded source tarball against SHA
>> checksum. Just for nay case like downloading error or malicious
>> inject. Did you try to say that binary package have no such safeguard?
>
> Exactly. The INDEX does not contain such information. The thing is to do
> that, the pointyhat INDEX format would have to differ from the ports
> INDEX format.
>

It could be renamed to PKGINDEX.

> A possiblity of course, but also a source of trouble if the INDEX format
> of the ports should ever change, something I desire:
> http://www.freebsd.org/cgi/query-pr.cgi?pr=148783
>

It's also missing ranged versions of dependencies like foo>=2.0.

> Another solution would be to add an empty column that pointyhat can fill
> in.

Actually I'd rather have less data in INDEX.
Some columns have no use in case of packages ({BUILD,FETCH,EXTRACT,PATCH}_DEPENDS).

Additional data can be stored in separate files as long as there is some way
to verify they all belong to same build. It could be some information
in header (does INDEX support comments?) or some assumption such as that
modification times may not differ by more than a couple of seconds
(just touch them all at the end of build).

I think storing all columns in separate files would be better and allow
future extensibility.
For example consider PKGINDEX subdirectory with following files,
each having one entry per line:

# PKGINDEX/format:
MANDATORY org.freebsd.PKGINDEX 1
OPTIONAL com.mycomany.PKGINDEX 1

# this one actually has custom format
# list base format and extensions
# mandatory must be supported by client, optional can be ignored
# i'm probably overengineering here a little ;)

# PKGINDEX/packages.list:
bar-2.0
baz-3.2
foo-1.0

# *.list files are just lists (ordered alphabetically)

# PKGINDEX/run-depends.map:
foo-1.0 bar-2.0
foo-1.0 baz>=3.1

# *.map files have "<key> <value>+" pairs (ordered alphabetically)
# in this case foo needs bar & baz
# in future it could support boolean operators
# e.g.:
# foo-1.0 (bar-2.0 || bar-alternate=2.0)
# foo-1.0 (baz>=3.1 && baz<4.0)
# where '-' is "soft" requirement like currently implemented in pkg_add
# new operators would be strictly enforced

# PKGINDEX/category.map:
foo-1.0 sysutils databases devel

# sysutils is primary category of foo-1.0
# additional categories follow in alphabetic order

# PKGINDEX/origin.map:
foo-1.0 sysutils/foo

# PKGINDEX/description.map:
bar-3.2 Bar is better than foo
foo-1.0 Foo is great

etc...
I have some awk scripts to produce something like that from INDEX ;)

General idea is to have sorted files where first column is always a key.
You can then operate on them using utilities like join(1) and tsort(1).

Having potentially large but rarely used attributes like description
in separate files greatly reduces amount of text needed to be loaded and
parsed by utilities in common operations.


_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: LICENSE questions
Next: deskutils/alexandria: maintainer timeout more than 1 month