dcrspy is a Go project for continuous monitoring and logging of Decred data from both the block chain server (dcrd) and optionally your wallet (dcrwallet), with an emphasis on PoS. https://github.com/chappjc/dcrspy dcrspy establishes a persistent websocket connection to both dcrd and dcrwallet, and registers a new block notifier with dcrd to trigger data collection and storage when a new block is processed by the daemon. Communication with dcrd and dcrwallet uses the Decred JSON-RPC API. Unlike dcrctl, which issues commands via HTTP POST, dcrspy remains connected (by default) using websockets. The initial motivation for dcrspy was to collect and format live data for a web server, but perhaps it's useful for any backend application or personal data monitoring on a desktop. There is a massive to-do list and as such is very much a work-in-progress. That said, I will try to keep master stable, developing new features in separate branches. Important acknowledgment: @ceejep and the dcrticketbuyer, which already has great live plotting of ticket prices, fees, and more. dcrspy does not have a web ui, nor one planned. Please see the current README.md and the release notes for the current release for more information about the latest features and developments. Data can be written in JSON format to the filesystem or stdout, or as a plain text summary similar to the following (wallet data redacted): Code: Block 35561: Stake difficulty: 22.663 -> 22.663 (current -> next block) Estimated price in next window: 25.279 / [24.63, 26.68] ([min, max]) Window progress: 138 / 144 of price window number 246 Ticket fees: 0.0101, 0.0101, 0.0000 (mean, median, std), n=1 Ticket pool: 42048 (size), 17.721 (avg. price), 745115.63 (total DCR locked) Wallet and Stake Info at Height 35561: - Balances Balances (spendable): xx.xxxx (default), xx.xxxx (all) Balances (locked): xxx.xxxx (default), xxxx.xxxx (all), xxxx.xxxx (imported) Balances (any): xxxx.xxxx (default), xxxx.xxxx (all) - Stake Info ===> Mining enabled: true; Unlocked: true <=== Mined tickets: 4 (immature), 43 (live) mempool tickets: 0 (own), 6 (all) Ticket price: 22.663 | Window progress: 138 / 144 Wallet's price: 23.8100; fee: 0.1940 / KiB Totals: 541 votes, 919.84 subsidy 1 missed, 1 revoked The JSON-formatted data is more thorough and easily parsed. For example, here's a just a piece of the block data: Code: { "estimatestakediff": { "min": 41.5196016, "max": 48.11919073, "expected": 48.06180299 }, "currentstakediff": { "current": 18.98413392, "next": 18.98413392 }, "ticketfeeinfo_block": { "height": 35832, "number": 20, "min": 0.4040404, "max": 0.4054054, "mean": 0.40465465, "median": 0.4040404, "stddev": 0.00069672 }, "block_header": { "hash": "00000000000001d9a4b01477e84bd12a13ad389a404fff80015e1ab93046b657", "confirmations": 1, "version": 1, ... Most data is obtainable with dcrctl, although some not so directly. Code: "ticket_pool_info": { "poolsize": 41813, "poolvalue": 747391.15346277, "poolvalavg": 17.87461204 } Code: "balances": { "allallacounts": 249.8002277, "alldefaultacount": 119.15575073, "spendableallaccounts": 9.60560466, "spendabledefaultaccount": 9.60560466, "lockedallaccounts": 239.46345289, "lockedimportedaccount": 173.69207521, "lockeddefaultaccount": 98.77137768 } (Balances above are fake).
The latest binary release is 0.7.0 with 0.7.3 on master (including lots of great changes). A version to support 0.8.0 and the getbalance changes is in the works. See the github releases page: https://github.com/chappjc/dcrspy/releases ------------------------------------------------------------------------- Latest released version (05 July 2016) v0.1.2 "Archer" This will be updated! https://github.com/chappjc/dcrspy/releases/tag/v0.1.2 The main features in this release (since 0.1.0) are: Email notifications for watched addresses. Mempool fee details (listing of the actual highest fees). Display a configurable window of fees in mempool to determine what is mineable in upcoming blocks. Lots of code commenting and updated documentation. IMPORTANT ERRATA: There's a bug when NOT using an email server, causing the error message "missing port in address". If none of the "watchaddress" flags request email notification with the ",1" suffix, it is safe to use a dummy email server and port like the one on sample-dcrspy.conf. This is fixed in the next version on master. I suggest to build it yourself with Go, but when it is released, you can download binaries for 64-bit Linux and Windows. ------------------------------------------------------------------------- Past spies: (27 Nov 2016) v0.7.3 not tagged (16 Nov 2016) v0.7.0 "Fletch" Decred JSON API v2.0.0 (16 Nov 2016) v0.6.0 "Millbarge" compatibility release (14 Sept 2016) v0.2.0 "Fitz-Hume" (05 July 2016) v0.1.2 "Archer" (21 June 2016) v0.1.1 (unreleased) (17 June 2016) v0.1.0 "Dick Steele" (12 June 2016) v0.0.3 "Johnny English" (09 June 2016) v0.0.2_1 "Jack Bauer"
Roadmap Next MySQL output (fixed table structure) Input and execute arbitrary system command on each new block (99% implemented, still considering generalizability) Added in v0.0.3, with height and hash substitutions. mempool monitoring (time.Ticker or notifier for newtx), important for ticket fees in first block of new price window Added in v0.1.0 Then Email notifications Added in v0.1.2 Display ticket fees in mempool to help get mined. Added in v0.1.1 List of addresses to watch (e.g. Your PoW reward address). Added in v0.1.0 Stake pool information (users, scripts, ticket histogram) MongoDB Useful ticket fee indicators. Started this in v0.1.2 with fee threshold for "minability" and ability to show window of sorted fees around this threshold, plus dumping all mempool fees. In-memory history for analysis (trending, temporal measures, etc.) Initial load of past block data Someday RESTful API Price analysis considering last N windows, to M windows in the future MySQL (user-defined structure)
dcrspy Failed to create data output folder /dcrspy/spydata/mainnet. Error: mkdir /dcrspy: permission denied looking through config.go it's not quite right.
Huh, config.go is mostly boilerplate from dcrticketbuyer and C0 stuff. What is your "outfolder" set to? It's supposed to default to [current directory]/spydata.
oh i see. had the windows outfolder in the sample config being used. looks like it created folders as needed now.
Ah, cool. I should have them both commented at the start. (EDIT: Fixed on master, along with typo in README.md) Anyway, it shouldn't write anything until the first block comes in (assuming you started with -j, --save-jsonfile). The initial summary is only written to stdout. BTW, I need to profile it because I think the pool size/value RPC calls are taking too long to be worth it.
up and running. very cool. i'll try and give notes/issues when I'm on vacay over the next few days. great job!
This is awesome! Built it and it seems to be working great. Getting this odd error in the log though: Code: 09:12:03 2016-06-13 [INF] DCRD: Block height 36627 connected 09:12:03 2016-06-13 [ERR] DSPY: Failed to start system command . Error: fork/exec : no such file or directory 09:12:03 2016-06-13 [INF] EXEC: Command execution complete (success). Any idea what it could be pointing at that I'm messing up? I did set up the .conf file using the template in the readme.
@Halestorm Hard to tell. Can you post both options, cmdname and cmdargs? It almost looks like cmdname is blank. (Blank is totally OK. No need to execute a command.) EDIT: OMG, my mistake. I left out the logic to skip command execution when there's no cmdname! Sorry. It should be a harmless warning that will be fixed in next version. Now fixed in master.
On master now (not release): Made ticket pool value optional since it takes close to 9 seconds. Now each block data query is typically less than 200ms. Also testing timeout on entire blockData.collect() call to prevent a hang if dcrd doesn't feel like responding.
Next version will include optional mempool monitoring so you can keep an eye on the fees in mempool in real time! For example, here is the output over several minutes at best block 37233: Code: Ticket fees (37233): 0.0518, 0.0203, 0.0571 (mean, median, std), n=2028 Ticket fees (37233): 0.0523, 0.0102, 0.0578 (mean, median, std), n=2033 Ticket fees (37233): 0.0524, 0.0253, 0.0580 (mean, median, std), n=2035 Ticket fees (37233): 0.0526, 0.1414, 0.0583 (mean, median, std), n=2038 Ticket fees (37233): 0.0526, 0.1419, 0.0583 (mean, median, std), n=2038 Ticket fees (37233): 0.0529, 0.0101, 0.0585 (mean, median, std), n=2041 Ticket fees (37233): 0.0529, 0.1717, 0.0585 (mean, median, std), n=2041 Ticket fees (37233): 0.0529, 0.1837, 0.0585 (mean, median, std), n=2041 Ticket fees (37233): 0.0529, 0.0103, 0.0585 (mean, median, std), n=2041 Ticket fees (37233): 0.0529, 0.2027, 0.0585 (mean, median, std), n=2041 Ticket fees (37233): 0.0533, 0.1520, 0.0590 (mean, median, std), n=2046 Ticket fees (37233): 0.0584, 0.0101, 0.0647 (mean, median, std), n=2112 Ticket fees (37233): 0.0584, 0.0101, 0.0647 (mean, median, std), n=2112 Ticket fees (37233): 0.0584, 0.0101, 0.0647 (mean, median, std), n=2112 Sadly, the median number is broken above since it is connected to a dcrd built prior to the recent median fix in dcrd.
Also added watched addresses (in development branch). For example, with a big pool's address in the config file: Code: watchaddress=DsZWrNNyKDUFPNMcjNYD7A8k9a4HCM5xgsW We get notified: Code: 18:19:49 2016-06-16 [INF] DSPY: Transaction with watched address DsZWrNNyKDUFPNMcjNYD7A8k9a4HCM5xgsW as outpoint mined into block 37543. The amount will be included.
Just released v0.1.0, nicknamed "Dick Steele". https://github.com/chappjc/dcrspy/releases/tag/v0.1.0 The main features in this release are mempool monitoring and watched addresses. Please see the release notes on GitHub for more information.
Wow, big compliments ! I tweaked your code to "Dick Steele +" to print 20 highest ticket fees in mempool - and I was able to actually "see" my tickets. Nice tool for people with limited time and money that want to purchase tickets manually with low fees. Nice !
Wow, that's planned for the next patch release already. Great idea! . Was actually thinking just the price of the 20th highest fee, but same idea. Thanks for the compliments! I think I might have some extra stats in addition to the top 20 fees, like the top quartile perhaps. I'll test out some ideas on a brach today. Along with a ton of corrections to the horrendous flag descriptions.
Disclaimer regarding mempool monitoring: The "mempool" is a data structure in your running daemon. If you just started dcrd, your mempool is basically cleared out. Even if there are a lot of tickets waiting to get mined, they are not waiting in your mempool if your daemon was not online when the ticket purchase transactions were inserted (do these tx get rebroadcast and at what frequency?). Thus, for accurate information, your daemon should be running since the start of the current price window. I believe this is correct, but I'd love if a developer could comment.
I am not sure about that "disclaimer", devs know better. Regarding my hack, it was just quick uncommenting your mempool stuff, please note this is more "c - like" hack, golang idiomatic would be to return the slice s[fromIdx:] Spoiler: ugly hack highest20 patch --- mempool.go.ori010 2016-06-20 17:51:57.577399398 +0200 +++ mempool.go 2016-06-20 17:51:57.577399398 +0200 @@ -146,15 +146,18 @@ // Just playing with getting all fees // mempoolTickets[ticketHashes[0].String()].Fee - // mempoolTickets, err := client.GetRawMempoolVerbose(dcrjson.GRMTickets) - // allFees := make([]float64, 0, len(ticketHashes)) - // for _, t := range mempoolTickets { - // txSize := float64(t.Size) - // allFees = append(allFees, t.Fee / txSize * 1000) - // } - // medianFee := MedianCoin(allFees) - // mempoolLog.Infof("Median fee computed: %v (%v)", medianFee, - // len(ticketHashes)) + mempoolTickets, err := client.GetRawMempoolVerbose(dcrjson.GRMTickets) + allFees := make([]float64, 0, len(ticketHashes)) + for _, t := range mempoolTickets { + txSize := float64(t.Size) + allFees = append(allFees, t.Fee / txSize * 1000) + } + medianFee := MedianCoin(allFees) + mempoolLog.Infof("Median fee computed: %v (%v)", medianFee, + len(ticketHashes)) + highest20 := make([]float64,0) + highest20coin(allFees,&highest20) + mempoolLog.Infof("Highest 20 fees in mempool: %v", highest20) // Decide if it is time to collect and record new data // 1. Get block height @@ -545,3 +548,17 @@ } return (s[middle] + s[middle-1]) / 2 } +func highest20coin(s []float64, slice* []float64) { + if len(s) < 1 { + return + } + + sort.Float64s(s) + + idxFrom := len(s) - 20 + if idxFrom < 0 { idxFrom = 0 } + + for i:= idxFrom; i< len(s); i++{ + *slice = append(*slice, s) + } +}