Mixing
Interpret foreign memory as lisp memory (or vice versa) without copying data
I try to write BLOB into database - chunk by chunk, using database API C-function (say, db-write-chunk).
This function takes a pointer to a foreign memory (where chunk is placed) as an argument.
So, I make buffer for a chunk: foreign-buffer.
I'll take chunk data from a file (or binary stream) by read-sequence into stream-buffer:
(let ((foreign-buffer (foreign-alloc :uchar 1024)))
(stream-buffer ((make-array 1024 :element-type '(unsigned-byte 8))))
(loop
for cnt = (read-sequence stream-buffer MY-STREAM)
while (> cnt 0)
do
;; copy cnt bytes from stream-buffer into foreign-buffer
;; call db-write-chunk with foreign-buffer
L in BLOB is for Large and loop may iterate many times.
Besides that, all this code may be wrapped by the external loop (bulk-insert, for example).
So, I want to minimize the count of steps in the loop(s) body.
To have this done I need:
to be able to read sequence not into stream-buffer, but into foreign-buffer directly, like this:
(read-sequence (coerce foreign-buffer '(vector/array ...)) MY-STREAM)
or to be able to interpret stream-buffer as foreign memory, like this:
(db-write-chunk (mem-aptr stream-buffer :uchar 0))
Is it possible to solve my problem using single buffer only - native or foreign, without copying memory between them?
common-lisp cffi
edited Nov 19 '18 at 20:46
sds
38.8k1493168
asked Nov 19 '18 at 20:12
MrCatMrCat
322
add a comment |
I try to write BLOB into database - chunk by chunk, using database API C-function (say, db-write-chunk).
This function takes a pointer to a foreign memory (where chunk is placed) as an argument.
So, I make buffer for a chunk: foreign-buffer.
I'll take chunk data from a file (or binary stream) by read-sequence into stream-buffer:
(let ((foreign-buffer (foreign-alloc :uchar 1024)))
(stream-buffer ((make-array 1024 :element-type '(unsigned-byte 8))))
(loop
for cnt = (read-sequence stream-buffer MY-STREAM)
while (> cnt 0)
do
;; copy cnt bytes from stream-buffer into foreign-buffer
;; call db-write-chunk with foreign-buffer
L in BLOB is for Large and loop may iterate many times.
Besides that, all this code may be wrapped by the external loop (bulk-insert, for example).
So, I want to minimize the count of steps in the loop(s) body.
To have this done I need:
to be able to read sequence not into stream-buffer, but into foreign-buffer directly, like this:
(read-sequence (coerce foreign-buffer '(vector/array ...)) MY-STREAM)
or to be able to interpret stream-buffer as foreign memory, like this:
(db-write-chunk (mem-aptr stream-buffer :uchar 0))
Is it possible to solve my problem using single buffer only - native or foreign, without copying memory between them?
common-lisp cffi
edited Nov 19 '18 at 20:46
sds
38.8k1493168
asked Nov 19 '18 at 20:12
MrCatMrCat
322
1
I think this is highly implementation-dependent. What Lisp are you using?
– sds
Nov 19 '18 at 20:46
3
maybe github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines
– David Hodge
Nov 20 '18 at 0:25
@sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL.
– MrCat
Nov 20 '18 at 13:55
@DavidHodge, thanks, I've just downloaded it and try to understand how it works
– MrCat
Nov 20 '18 at 13:58
1
Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled).
– MrCat
Nov 20 '18 at 15:45
add a comment |
I try to write BLOB into database - chunk by chunk, using database API C-function (say, db-write-chunk).
This function takes a pointer to a foreign memory (where chunk is placed) as an argument.
So, I make buffer for a chunk: foreign-buffer.
I'll take chunk data from a file (or binary stream) by read-sequence into stream-buffer:
(let ((foreign-buffer (foreign-alloc :uchar 1024)))
(stream-buffer ((make-array 1024 :element-type '(unsigned-byte 8))))
(loop
for cnt = (read-sequence stream-buffer MY-STREAM)
while (> cnt 0)
do
;; copy cnt bytes from stream-buffer into foreign-buffer
;; call db-write-chunk with foreign-buffer
L in BLOB is for Large and loop may iterate many times.
Besides that, all this code may be wrapped by the external loop (bulk-insert, for example).
So, I want to minimize the count of steps in the loop(s) body.
To have this done I need:
to be able to read sequence not into stream-buffer, but into foreign-buffer directly, like this:
(read-sequence (coerce foreign-buffer '(vector/array ...)) MY-STREAM)
or to be able to interpret stream-buffer as foreign memory, like this:
(db-write-chunk (mem-aptr stream-buffer :uchar 0))
Is it possible to solve my problem using single buffer only - native or foreign, without copying memory between them?
common-lisp cffi
edited Nov 19 '18 at 20:46
sds
38.8k1493168
asked Nov 19 '18 at 20:12
MrCatMrCat
322
I try to write BLOB into database - chunk by chunk, using database API C-function (say, db-write-chunk).
This function takes a pointer to a foreign memory (where chunk is placed) as an argument.
So, I make buffer for a chunk: foreign-buffer.
I'll take chunk data from a file (or binary stream) by read-sequence into stream-buffer:
(let ((foreign-buffer (foreign-alloc :uchar 1024)))
(stream-buffer ((make-array 1024 :element-type '(unsigned-byte 8))))
(loop
for cnt = (read-sequence stream-buffer MY-STREAM)
while (> cnt 0)
do
;; copy cnt bytes from stream-buffer into foreign-buffer
;; call db-write-chunk with foreign-buffer
L in BLOB is for Large and loop may iterate many times.
Besides that, all this code may be wrapped by the external loop (bulk-insert, for example).
So, I want to minimize the count of steps in the loop(s) body.
To have this done I need:
to be able to read sequence not into stream-buffer, but into foreign-buffer directly, like this:
(read-sequence (coerce foreign-buffer '(vector/array ...)) MY-STREAM)
or to be able to interpret stream-buffer as foreign memory, like this:
(db-write-chunk (mem-aptr stream-buffer :uchar 0))
Is it possible to solve my problem using single buffer only - native or foreign, without copying memory between them?
common-lisp cffi
common-lisp cffi
edited Nov 19 '18 at 20:46
sds
38.8k1493168
asked Nov 19 '18 at 20:12
MrCatMrCat
322
edited Nov 19 '18 at 20:46
sds
38.8k1493168
asked Nov 19 '18 at 20:12
MrCatMrCat
322
edited Nov 19 '18 at 20:46
sds
38.8k1493168
edited Nov 19 '18 at 20:46
sds
38.8k1493168
edited Nov 19 '18 at 20:46
sds
38.8k1493168
38.8k1493168
asked Nov 19 '18 at 20:12
MrCatMrCat
322
asked Nov 19 '18 at 20:12
MrCatMrCat
322
asked Nov 19 '18 at 20:12
MrCatMrCat
322
322
1
I think this is highly implementation-dependent. What Lisp are you using?
– sds
Nov 19 '18 at 20:46
3
maybe github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines
– David Hodge
Nov 20 '18 at 0:25
@sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL.
– MrCat
Nov 20 '18 at 13:55
@DavidHodge, thanks, I've just downloaded it and try to understand how it works
– MrCat
Nov 20 '18 at 13:58
1
Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled).
– MrCat
Nov 20 '18 at 15:45
add a comment |
1
I think this is highly implementation-dependent. What Lisp are you using?
– sds
Nov 19 '18 at 20:46
3
maybe github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines
– David Hodge
Nov 20 '18 at 0:25
@sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL.
– MrCat
Nov 20 '18 at 13:55
@DavidHodge, thanks, I've just downloaded it and try to understand how it works
– MrCat
Nov 20 '18 at 13:58
1
Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled).
– MrCat
Nov 20 '18 at 15:45
1
1
I think this is highly implementation-dependent. What Lisp are you using?
– sds
Nov 19 '18 at 20:46
I think this is highly implementation-dependent. What Lisp are you using?
– sds
Nov 19 '18 at 20:46
3
3
maybe github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines
– David Hodge
Nov 20 '18 at 0:25
maybe github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines
– David Hodge
Nov 20 '18 at 0:25
@sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL.
– MrCat
Nov 20 '18 at 13:55
@sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL.
– MrCat
Nov 20 '18 at 13:55
@DavidHodge, thanks, I've just downloaded it and try to understand how it works
– MrCat
Nov 20 '18 at 13:58
@DavidHodge, thanks, I've just downloaded it and try to understand how it works
– MrCat
Nov 20 '18 at 13:58
1
1
Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled).
– MrCat
Nov 20 '18 at 15:45
Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled).
– MrCat
Nov 20 '18 at 15:45
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53381958%2finterpret-foreign-memory-as-lisp-memory-or-vice-versa-without-copying-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53381958%2finterpret-foreign-memory-as-lisp-memory-or-vice-versa-without-copying-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
I think this is highly implementation-dependent. What Lisp are you using?
– sds
Nov 19 '18 at 20:46
3
maybe github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines
– David Hodge
Nov 20 '18 at 0:25
@sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL.
– MrCat
Nov 20 '18 at 13:55
@DavidHodge, thanks, I've just downloaded it and try to understand how it works
– MrCat
Nov 20 '18 at 13:58
1
Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled).
– MrCat
Nov 20 '18 at 15:45