Decred Monitor And Data Collector - Dcrspy

Discussion in 'User Projects' started by chappjc, Jun 10, 2016.

Tags:
  1. 2017/12/15 - Decred v1.1.2 released! → Release Notes  → Downloads
  1. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #1 chappjc, Jun 10, 2016
    Last edited: Jun 21, 2016
    dcrspy is a Go project for continuous monitoring and logging of Decred data from both the block chain server (dcrd) and optionally your wallet (dcrwallet), with an emphasis on PoS.

    https://github.com/chappjc/dcrspy

    dcrspy establishes a persistent websocket connection to both dcrd and dcrwallet, and registers a new block notifier with dcrd to trigger data collection and storage when a new block is processed by the daemon. Communication with dcrd and dcrwallet uses the Decred JSON-RPC API. Unlike dcrctl, which issues commands via HTTP POST, dcrspy remains connected (by default) using websockets.

    The initial motivation for dcrspy was to collect and format live data for a web server, but perhaps it's useful for any backend application or personal data monitoring on a desktop. There is a massive to-do list and as such is very much a work-in-progress. That said, I will try to keep master stable, developing new features in separate branches.

    Important acknowledgment: @ceejep and the dcrticketbuyer, which already has great live plotting of ticket prices, fees, and more. dcrspy does not have a web ui, nor one planned.

    Please see the current README.md and the release notes for the current release for more information about the latest features and developments.

    Data can be written in JSON format to the filesystem or stdout, or as a plain text summary similar to the following (wallet data redacted):

    Code:
    Block 35561:
           Stake difficulty: 22.663 -> 22.663 (current -> next block)
           Estimated price in next window: 25.279 / [24.63, 26.68] ([min, max])
           Window progress: 138 / 144 of price window number 246
           Ticket fees: 0.0101, 0.0101, 0.0000 (mean, median, std), n=1
           Ticket pool: 42048 (size), 17.721 (avg. price), 745115.63 (total DCR locked)
    
    Wallet and Stake Info at Height 35561:
    - Balances
           Balances (spendable): xx.xxxx (default), xx.xxxx (all)
           Balances (locked): xxx.xxxx (default), xxxx.xxxx (all), xxxx.xxxx (imported)
           Balances (any): xxxx.xxxx (default), xxxx.xxxx (all)
    - Stake Info
           ===> Mining enabled: true; Unlocked: true <===
           Mined tickets: 4 (immature), 43 (live)
           mempool tickets: 0 (own), 6 (all)
           Ticket price: 22.663 | Window progress: 138 / 144
           Wallet's price: 23.8100; fee: 0.1940 / KiB
           Totals: 541 votes, 919.84 subsidy
           1 missed, 1 revoked
    
    The JSON-formatted data is more thorough and easily parsed. For example, here's a just a piece of the block data:

    Code:
    {
        "estimatestakediff": {
            "min": 41.5196016,
            "max": 48.11919073,
            "expected": 48.06180299
        },
        "currentstakediff": {
            "current": 18.98413392,
            "next": 18.98413392
        },
        "ticketfeeinfo_block": {
            "height": 35832,
            "number": 20,
            "min": 0.4040404,
            "max": 0.4054054,
            "mean": 0.40465465,
            "median": 0.4040404,
            "stddev": 0.00069672
        },
        "block_header": {
            "hash": "00000000000001d9a4b01477e84bd12a13ad389a404fff80015e1ab93046b657",
            "confirmations": 1,
            "version": 1,
        ...
    
    Most data is obtainable with dcrctl, although some not so directly.

    Code:
    "ticket_pool_info": {
        "poolsize": 41813,
        "poolvalue": 747391.15346277,
        "poolvalavg": 17.87461204
    }
    
    Code:
    "balances": {
        "allallacounts": 249.8002277,
        "alldefaultacount": 119.15575073,
        "spendableallaccounts": 9.60560466,
        "spendabledefaultaccount": 9.60560466,
        "lockedallaccounts": 239.46345289,
        "lockedimportedaccount": 173.69207521,
        "lockeddefaultaccount": 98.77137768
    }
    
    (Balances above are fake).
     
    zero, Shadowlance, Halestorm and 4 others like this.
  2. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #2 chappjc, Jun 10, 2016
    Last edited: Jan 27, 2017
    The latest binary release is 0.7.0 with 0.7.3 on master (including lots of great changes). A version to support 0.8.0 and the getbalance changes is in the works. See the github releases page:

    https://github.com/chappjc/dcrspy/releases

    -------------------------------------------------------------------------

    Latest released version (05 July 2016) v0.1.2 "Archer" This will be updated!

    https://github.com/chappjc/dcrspy/releases/tag/v0.1.2

    The main features in this release (since 0.1.0) are:
    • Email notifications for watched addresses.
    • Mempool fee details (listing of the actual highest fees). Display a configurable window of fees in mempool to determine what is mineable in upcoming blocks.
    • Lots of code commenting and updated documentation.
    IMPORTANT ERRATA: There's a bug when NOT using an email server, causing the error message "missing port in address". If none of the "watchaddress" flags request email notification with the ",1" suffix, it is safe to use a dummy email server and port like the one on sample-dcrspy.conf. This is fixed in the next version on master.

    I suggest to build it yourself with Go, but when it is released, you can download binaries for 64-bit Linux and Windows.

    -------------------------------------------------------------------------

    Past spies:

    (27 Nov 2016) v0.7.3 not tagged
    (16 Nov 2016) v0.7.0 "Fletch" Decred JSON API v2.0.0
    (16 Nov 2016) v0.6.0 "Millbarge" compatibility release
    (14 Sept 2016) v0.2.0 "Fitz-Hume"
    (05 July 2016) v0.1.2 "Archer"
    (21 June 2016) v0.1.1 (unreleased)
    (17 June 2016) v0.1.0 "Dick Steele"
    (12 June 2016) v0.0.3 "Johnny English"
    (09 June 2016) v0.0.2_1 "Jack Bauer"
     
  3. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #3 chappjc, Jun 10, 2016
    Last edited: Jul 5, 2016
    Roadmap

    Next
    • MySQL output (fixed table structure)
    • Input and execute arbitrary system command on each new block (99% implemented, still considering generalizability) Added in v0.0.3, with height and hash substitutions.
    • mempool monitoring (time.Ticker or notifier for newtx), important for ticket fees in first block of new price window Added in v0.1.0

    Then
    • Email notifications Added in v0.1.2
    • Display ticket fees in mempool to help get mined. Added in v0.1.1
    • List of addresses to watch (e.g. Your PoW reward address). Added in v0.1.0
    • Stake pool information (users, scripts, ticket histogram)
    • MongoDB
    • Useful ticket fee indicators. Started this in v0.1.2 with fee threshold for "minability" and ability to show window of sorted fees around this threshold, plus dumping all mempool fees.
    • In-memory history for analysis (trending, temporal measures, etc.)
    • Initial load of past block data

    Someday
    • RESTful API
    • Price analysis considering last N windows, to M windows in the future
    • MySQL (user-defined structure)
     
    jy-p likes this.
  4. ay-p

    ay-p Full Member
    Developer

    Dec 7, 2015
    148
    106
    Male
    looks very cool. taking a look through it all now.
     
  5. ay-p

    ay-p Full Member
    Developer

    Dec 7, 2015
    148
    106
    Male
    dcrspy
    Failed to create data output folder /dcrspy/spydata/mainnet. Error: mkdir /dcrspy: permission denied

    looking through config.go it's not quite right.
     
  6. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    Huh, config.go is mostly boilerplate from dcrticketbuyer and C0 stuff.

    What is your "outfolder" set to? It's supposed to default to [current directory]/spydata.
     
  7. ay-p

    ay-p Full Member
    Developer

    Dec 7, 2015
    148
    106
    Male
    oh i see. had the windows outfolder in the sample config being used.

    looks like it created folders as needed now.
     
  8. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    Ah, cool. I should have them both commented at the start. (EDIT: Fixed on master, along with typo in README.md)

    Anyway, it shouldn't write anything until the first block comes in (assuming you started with -j, --save-jsonfile). The initial summary is only written to stdout.

    BTW, I need to profile it because I think the pool size/value RPC calls are taking too long to be worth it.
     
    ay-p likes this.
  9. ay-p

    ay-p Full Member
    Developer

    Dec 7, 2015
    148
    106
    Male
    up and running. very cool. i'll try and give notes/issues when I'm on vacay over the next few days.

    great job!
     
    chappjc likes this.
  10. Halestorm

    Halestorm Jr. Member

    Jan 22, 2016
    83
    32
    Male
    Data Center
    Texas, USA
    This is awesome! Built it and it seems to be working great. Getting this odd error in the log though:

    Code:
    09:12:03 2016-06-13 [INF] DCRD: Block height 36627 connected
    09:12:03 2016-06-13 [ERR] DSPY: Failed to start system command . Error: fork/exec : no such file or directory
    09:12:03 2016-06-13 [INF] EXEC: Command execution complete (success).
    Any idea what it could be pointing at that I'm messing up? I did set up the .conf file using the template in the readme.
     
  11. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #11 chappjc, Jun 13, 2016
    Last edited: Jun 13, 2016
    @Halestorm Hard to tell. Can you post both options, cmdname and cmdargs? It almost looks like cmdname is blank. (Blank is totally OK. No need to execute a command.)

    EDIT: OMG, my mistake. I left out the logic to skip command execution when there's no cmdname! Sorry. It should be a harmless warning that will be fixed in next version. Now fixed in master.
     
    Halestorm likes this.
  12. Halestorm

    Halestorm Jr. Member

    Jan 22, 2016
    83
    32
    Male
    Data Center
    Texas, USA
    @chappjc Sweet. :D (Yay I contributed!)
     
    chappjc likes this.
  13. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #13 chappjc, Jun 14, 2016
    Last edited: Jun 14, 2016
    On master now (not release): Made ticket pool value optional since it takes close to 9 seconds. Now each block data query is typically less than 200ms. Also testing timeout on entire blockData.collect() call to prevent a hang if dcrd doesn't feel like responding.
     
  14. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #14 chappjc, Jun 15, 2016
    Last edited: Jun 15, 2016
    Next version will include optional mempool monitoring so you can keep an eye on the fees in mempool in real time!

    For example, here is the output over several minutes at best block 37233:

    Code:
    Ticket fees (37233):  0.0518, 0.0203, 0.0571 (mean, median, std), n=2028
    Ticket fees (37233):  0.0523, 0.0102, 0.0578 (mean, median, std), n=2033
    Ticket fees (37233):  0.0524, 0.0253, 0.0580 (mean, median, std), n=2035
    Ticket fees (37233):  0.0526, 0.1414, 0.0583 (mean, median, std), n=2038
    Ticket fees (37233):  0.0526, 0.1419, 0.0583 (mean, median, std), n=2038
    Ticket fees (37233):  0.0529, 0.0101, 0.0585 (mean, median, std), n=2041
    Ticket fees (37233):  0.0529, 0.1717, 0.0585 (mean, median, std), n=2041
    Ticket fees (37233):  0.0529, 0.1837, 0.0585 (mean, median, std), n=2041
    Ticket fees (37233):  0.0529, 0.0103, 0.0585 (mean, median, std), n=2041
    Ticket fees (37233):  0.0529, 0.2027, 0.0585 (mean, median, std), n=2041
    Ticket fees (37233):  0.0533, 0.1520, 0.0590 (mean, median, std), n=2046
    Ticket fees (37233):  0.0584, 0.0101, 0.0647 (mean, median, std), n=2112
    Ticket fees (37233):  0.0584, 0.0101, 0.0647 (mean, median, std), n=2112
    Ticket fees (37233):  0.0584, 0.0101, 0.0647 (mean, median, std), n=2112
    
    Sadly, the median number is broken above since it is connected to a dcrd built prior to the recent median fix in dcrd.
     
    David likes this.
  15. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    Also added watched addresses (in development branch). For example, with a big pool's address in the config file:

    Code:
    watchaddress=DsZWrNNyKDUFPNMcjNYD7A8k9a4HCM5xgsW
    We get notified:

    Code:
    18:19:49 2016-06-16 [INF] DSPY: Transaction with watched address DsZWrNNyKDUFPNMcjNYD7A8k9a4HCM5xgsW as outpoint mined into block 37543.
    The amount will be included. :)
     
  16. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    root likes this.
  17. root

    root Member

    Feb 3, 2016
    381
    76
    Wow, big compliments !
    I tweaked your code to "Dick Steele +" to print 20 highest ticket fees in mempool - and I was able to actually "see" my tickets. Nice tool for people with limited time and money that want to purchase tickets manually with low fees.
    Nice !
     
    chappjc likes this.
  18. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #18 chappjc, Jun 20, 2016
    Last edited: Jun 20, 2016
    Wow, that's planned for the next patch release already. Great idea! ;). Was actually thinking just the price of the 20th highest fee, but same idea.

    Thanks for the compliments!

    I think I might have some extra stats in addition to the top 20 fees, like the top quartile perhaps. I'll test out some ideas on a brach today. Along with a ton of corrections to the horrendous flag descriptions.
     
  19. chappjc

    chappjc Full Member
    Developer Pool Operator (PoS)

    #19 chappjc, Jun 20, 2016
    Last edited: Jun 20, 2016
    Disclaimer regarding mempool monitoring: The "mempool" is a data structure in your running daemon. If you just started dcrd, your mempool is basically cleared out. Even if there are a lot of tickets waiting to get mined, they are not waiting in your mempool if your daemon was not online when the ticket purchase transactions were inserted (do these tx get rebroadcast and at what frequency?). Thus, for accurate information, your daemon should be running since the start of the current price window. I believe this is correct, but I'd love if a developer could comment.
     
  20. root

    root Member

    Feb 3, 2016
    381
    76
    I am not sure about that "disclaimer", devs know better.
    Regarding my hack, it was just quick uncommenting your mempool stuff, please note this is more "c - like" hack, golang idiomatic would be to return the slice s[fromIdx:]
    --- mempool.go.ori010 2016-06-20 17:51:57.577399398 +0200
    +++ mempool.go 2016-06-20 17:51:57.577399398 +0200
    @@ -146,15 +146,18 @@

    // Just playing with getting all fees
    // mempoolTickets[ticketHashes[0].String()].Fee
    - // mempoolTickets, err := client.GetRawMempoolVerbose(dcrjson.GRMTickets)
    - // allFees := make([]float64, 0, len(ticketHashes))
    - // for _, t := range mempoolTickets {
    - // txSize := float64(t.Size)
    - // allFees = append(allFees, t.Fee / txSize * 1000)
    - // }
    - // medianFee := MedianCoin(allFees)
    - // mempoolLog.Infof("Median fee computed: %v (%v)", medianFee,
    - // len(ticketHashes))
    + mempoolTickets, err := client.GetRawMempoolVerbose(dcrjson.GRMTickets)
    + allFees := make([]float64, 0, len(ticketHashes))
    + for _, t := range mempoolTickets {
    + txSize := float64(t.Size)
    + allFees = append(allFees, t.Fee / txSize * 1000)
    + }
    + medianFee := MedianCoin(allFees)
    + mempoolLog.Infof("Median fee computed: %v (%v)", medianFee,
    + len(ticketHashes))
    + highest20 := make([]float64,0)
    + highest20coin(allFees,&highest20)
    + mempoolLog.Infof("Highest 20 fees in mempool: %v", highest20)

    // Decide if it is time to collect and record new data
    // 1. Get block height
    @@ -545,3 +548,17 @@
    }
    return (s[middle] + s[middle-1]) / 2
    }
    +func highest20coin(s []float64, slice* []float64) {
    + if len(s) < 1 {
    + return
    + }
    +
    + sort.Float64s(s)
    +
    + idxFrom := len(s) - 20
    + if idxFrom < 0 { idxFrom = 0 }
    +
    + for i:= idxFrom; i< len(s); i++{
    + *slice = append(*slice, s)
    + }
    +}

     

Share This Page