(12:51:29) andyjpb: i.e. they're slow and the cpu load is low
(12:56:47) zaniyah: disk I/O?
(12:57:08) zaniyah: or is it a lot of network I/O?  I've had a network port saturated before.
zanea|away zaniyah 
(12:57:44) andyjpb: zaniyah: loopback interface
(12:57:58) andyjpb: can one saturate the loopback interface?
(12:58:03) C-Keen: read-file takes an arbitrary reader procedure
(12:58:16) zaniyah: andyjpb: I wouldn't think so
(12:58:16) andyjpb: what did iterrogo optimize? read-u8vector takes 0.68s on my test benchmark
(12:58:29) zaniyah: I don't think it uses the PCI bus either so that can't be the bottleneck
(12:58:34) andyjpb: compared with 0.124s with (string->blob (read-string ...))
(13:00:03) zaniyah: andyjpb: what's the java doing when queried?  Does it look something up in a hashmap or anything like that (assume a database isn't involved and it isn't doing reads from disk)?
(13:00:08) andyjpb: also, despite the doc page saying to pass #f for an unspecified length, I get
(13:00:10) andyjpb:   (stargate-lolevel.scm:130) in procedure call to `read-u8vector', expected argument #1 of type `fixnum', but was given an argument of type `false'
(13:00:15) andyjpb: http://api.call-cc.org/doc/srfi-4
zanea|away zaniyah 
(13:00:28) andyjpb: zaniyah: it's hbase via the stargate rest api
(13:00:36) andyjpb: my next check will be to benchmark the api with curl
(13:00:41) andyjpb: rather than my chicken progam
(13:00:47) andyjpb: if curl is slow then I'll accept it
(13:01:28) sjamaan left the room (quit: Remote host closed the connection).
(13:01:59) zaniyah: andyjpb: oh, hbase ..
(13:02:08) andyjpb: I think it's somewhere else
(13:02:17) andyjpb: shell's time for my benchmark claims 5s
(13:02:22) zaniyah: do you have no way to profile it?
(13:02:25) andyjpb: but chicken's time claims 0.12
(13:02:35) zaniyah: /usr/bin/time ?
(13:02:36) andyjpb: so it might be on the other end of the rest api
(13:02:41) andyjpb: $ time csi -s extractor.scm
(13:02:41) andyjpb: 0.136s CPU time, 0.02s GC time (major), 27403/4634 mutations (total/tracked), 4/213 GCs (major/minor)
(13:02:41) andyjpb: real    0m5.269s
(13:02:41) andyjpb: user    0m0.344s
(13:02:41) andyjpb: sys     0m0.024s
(13:02:41) zaniyah: the shell's built in isn't as good
(13:02:58) andyjpb: $ time ./extractor 
(13:02:58) andyjpb: 0.12s CPU time, 0.016s GC time (major), 27282/4712 mutations (total/tracked), 4/213 GCs (major/minor)
(13:02:58) andyjpb: real    0m5.082s
(13:02:58) andyjpb: user    0m0.152s
(13:02:58) andyjpb: sys     0m0.016s
(13:03:22) andyjpb: and when I attach gdb to the chicken process I see it waiting in poll a lot
(13:04:33) ***andyjpb breaks out curl
(13:15:00) C-Keen: andyjpb: that's the scheduler
(13:18:55) ovenpasta left the room (quit: Ping timeout: 248 seconds).
(13:27:41) sjamaan [~sjamaan@netbsd/developer/sjamaan] entered the room.
(13:28:47) sjamaan: andyjpb: Did I miss anything after "i.e. they're slow and the cpu load is low"?
(13:29:03) sjamaan: My hosting provider is having some issues on their network
(13:32:34) sjamaan: andyjpb: u8vectors are blobs wrapped up into a record type
(13:33:06) sjamaan: (that's why blob->u8vector/shared works; it just creates a 2-slot record with the type and the existing blob)
(13:38:40) Youbi left the room (quit: Quit: WeeChat 1.0).
(14:00:57) andyjpb: sjamaan: weird. u8vectors are about 5 or 6 times slower than blobs