Prev: Postgresql 8.3 issue
Next: Auto-explain patch
From: "Stephen R. van den Berg" on 30 Jun 2008 07:17
I'm looking at the most efficient and lean way to interface with the
DB in a least-overhead scenario to process large(r) amounts of binary
data.

For simplicity, I want to avoid using the Large-Object facility.

It seems that the most efficient way to communicate with the DB would
be through PQexecParams(), which avoids the whole bytea-encoding issues.

However, two questions spring to mind:

- The docs say that you can use $1, $2, etc. to reference parameters.
What happens if you have more than 9 parameters?
Does it become $10 or ${10} or $(10) or is it simply not possible
te reference more than nine parameters this way?

- Say that the SELECT returns 1000 rows of 100MB each, is there a way
to avoid PQexecParams() from wanting to allocate 1000*100MB = 100GB
at once, and somehow extract the rows in smaller chunks?
(Incidentally, MySQL has such a facility).
I.e. we call libpq several times, and get a few rows at a time, which
are read from the DB-stream when needed.
--
Sincerely,
Stephen R. van den Berg.

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Abhijit Menon-Sen on 30 Jun 2008 07:25
At 2008-06-30 13:17:42 +0200, srb(a)cuci.nl wrote:
>
> It seems that the most efficient way to communicate with the
> DB would be through PQexecParams(), which avoids the whole
> bytea-encoding issues.

Yes.

> Does it become $10 or ${10} or $(10) or is it simply not possible
> te reference more than nine parameters this way?

$10 etc.

> - Say that the SELECT returns 1000 rows of 100MB each, is there a way
> to avoid PQexecParams() from wanting to allocate 1000*100MB = 100GB

Use a cursor and keep executing FETCH.

-- ams

--
Sent via pgsql-hackers mailing list (pgsql-hackers(a)postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

 | 
Pages: 1
Prev: Postgresql 8.3 issue
Next: Auto-explain patch