doc: update dchat tutorial chapter 1

doc/src/learn/dchat/dchat.md

@@ -2,15 +2,14 @@
 
 This tutorial will teach you how to deploy an app on DarkFi's p2p network.
 
-We will create a terminal-based p2p chat app called dchat that we run
-in two different instances: an inbound and outbound node called Alice
-and Bob. Alice takes a message from `stdin` and broadcasts it to the
-p2p network. When Bob receives the message on the p2p network it is
-displayed in his terminal.
+We will create a terminal-based p2p chat app called dchat. The chat app
+has two parts: a p2p daemon called `dchatd` and a python command-line
+tool for interacting with the daemon called `dchat-cli`.
 
 Dchat will showcase some key concepts that you'll need to develop on
 the p2p network, in particular:
 
+* Creating a p2p daemon.
 * Understanding inbound, outbound and seed nodes.
 * Writing and registering a custom `Protocol`.
 * Creating and subscribing to a custom `Message` type.
@@ -18,3 +17,4 @@ the p2p network, in particular:
 The source code for this tutorial can be found at
 [example/dchat](https://github.com/darkrenaissance/darkfi/tree/master/example/dchat).
 
+TODO: `dchat-cli` is currently only partially implemented.

doc/src/learn/dchat/deployment/deploy.md

@@ -1,8 +1,8 @@
 # Deploying a local network
 
-Get ready to spin up a bunch of different terminals. We are going to
-run 3 nodes: Alice and Bob and our seed node. To run the seed node,
-go to the `lilith` directory and spawn a new config file by running it once:
+Let's start by running 2 nodes: a `dchatd` full node and our seed node. To
+run the seed node, go to the `lilith` directory and spawn a new config
+file by running it once:
 
 ```bash
 cd darkfi
@@ -16,13 +16,15 @@ You should see the following output:
 Config file created in '"/home/USER/.config/darkfi/lilith_config.toml"'. Please review it and try again.
  ```
 
-Add dchat to the config as follows, keeping in mind that the port number must match the seed we specified
-earlier in Alice and Bob's settings.
+Add dchat to the config as follows, keeping in mind that the port number
+must match the seed we specified earlier in the TOML.
 
 ```toml
 [network."dchat"]
-port = 50515
-localnet = true
+accept_addrs = ["tcp://127.0.0.1:50515"]
+seeds = []
+peers = []
+version = "0.4.2"
 ```
 
 Now run `lilith`:
@@ -41,27 +43,19 @@ Here's what the debug output should look like:
 [INFO] Starting 0 outbound connection slots.
 ```
 
-Next we'll head back to `dchat` and run Alice. 
+Next we'll run `dchatd` outbound and inbound node using the default
+settings we specified earlier.
 
 ```bash
-cargo run a
+./dchatd
 ```
 
-You can `cat` or `tail` the log file created in /tmp/. I recommend using
-multitail for colored debug output, like so:
-
-```bash
-multitail -c /tmp/alice.log
 ```
-
-Check out that debug output! Keep an eye out for this line:
-
-```
-[INFO] Connected seed #0 [tcp://127.0.0.1:55555]
+[INFO] Connected seed #0 [tcp://127.0.0.1:50515]
 ```
 
-That shows Alice has connected to the seed node. Here's some more
-interesting output:
+That shows we have connected connected to the seed node. Here's some
+more interesting output:
 
 ```
 [DEBUG] (1) net: Attached ProtocolPing
@@ -70,26 +64,10 @@ interesting output:
 [DEBUG] (1) net: ProtocolVersion::exchange_versions() [START]
 ```
 
-This raises an interesting question- what are these protocols? We'll deal
-with that in more detail in a subsequent section. For now it's worth
-noting that every node on the p2p network performs several protocols
-when it connects to another node.
-
-Keep Alice and the seed node running. Now let's run Bob.
-
-```bash
-cargo run b
-```
-
-And track his debug output:
-
-```bash
-multitail -c /tmp/bob.log
-```
-
-Success! All going well, Alice and Bob are now connected to each
-other. We should be able to watch `ping` and `pong` messages being sent
-across by tracking their debug output.
+We have created a local deployment of the p2p network. 
 
-We have created a local deployment of the p2p network.
+This raises an interesting question- what are these protocols? We'll
+deal with that in more detail soon. For now it's worth noting that every
+node on the p2p network performs several protocols when it connects to
+another node.
 

doc/src/learn/dchat/deployment/seed-node.md

@@ -1,77 +1,42 @@
 # The seed node
 
-Let's create an instance of dchat inside our main function and pass the
-p2p network into it.  Then we'll add `dchat::start()` to our async loop
-in the main function. 
+Let's try building `dchatd` at this point and running it. Assuming
+`dchatd` is located in the `example/dchat` directory, we build it from
+the `darkfi` root directory using the following command:
 
-```rust
-#[async_std::main]
-async fn main() -> Result<()> {
-    let settings: Result<Settings> = match std::env::args().nth(1) {
-        Some(id) => match id.as_str() {
-            "a" => alice(),
-            "b" => bob(),
-            _ => Err(MissingSpecifier.into()),
-        },
-        None => Err(MissingSpecifier.into()),
-    };
-
-    let p2p = net::P2p::new(settings?.into()).await;
-
-    let dchat = Dchat::new(p2p);
-
-    let nthreads = std::thread::available_parallelism().unwrap().get();
-    let (signal, shutdown) = async_channel::unbounded::<()>();
+```bash
+cargo build --all-features --package dchatd
+```
 
-    let ex = Arc::new(Executor::new());
-    let ex2 = ex.clone();
+On first run, it will create a config file from the defaults we specified
+earlier. Run it as follows:
 
-    let (_, result) = Parallel::new()
-        .each(0..nthreads, |_| {
-            smol::future::block_on(ex.run(shutdown.recv()))
-        })
-        .finish(|| {
-            smol::future::block_on(async move {
-                dchat.start(ex2).await?;
-                drop(signal);
-                Ok(())
-            })
-        });
+```bash`
+./dchatd
 
-    result
-}
 ```
-
-Now try to run the program, don't forget to add a specifier `a` or `b`
-to define the type of node.
-
-It should output the following error: 
+It should output the following:
 
 ```
-Error: NetworkOperationFailed
+[WARN] [P2P] Failure contacting seed #0 [tcp://127.0.0.1:50515]: IO error: connection refused
+[WARN] [P2P] Seed #0 connection failed: IO error: connection refused
+[ERROR] [P2P] Network reseed failed: Failed to reach any seeds
 ```
 
-That's because there is no seed node online for our nodes to connect to. A
+That's because there is no seed node online for our node to connect to. A
 seed node is used when connecting to the network: it is a special kind
 of inbound node that gets connected to, sends over a list of addresses
 and disconnects again.  This behavior is defined in the `ProtocolSeed`.
 
-Everytime we run `p2p.start()` we attempt to connect to a seed using a
-`SeedSyncSession`.  If the `SeedSyncSession` fails, `p2p.start()` will fail,
-so without a seed node, our inbound and outbound nodes cannot establish
-a connection to the network. Let's remedy that.
-
-We have two options here. First, we could implement our own seed node.
-Alternatively, DarkFi maintains a master seed node called `lilith` that
-can act as the seed for many different protocols at the same time. For
-the purpose of this tutorial let's use `lilith`.
-
-What `lilith` does in the background is very simple. Just like any p2p
-daemon, a seed node defines its networks settings into a type called
-`Settings` and creates a new instance of the p2p network. It then runs
-`p2p::start()` and `p2p::run()`. The difference is in the settings: a seed
-node just specifies an inbound address which other nodes will connect to.
+Everytime we start an `OutboundSession`, we attempt to connect to a seed
+using a `SeedSyncSession`.  If the `SeedSyncSession` fails we cannot
+establish any outbound connections. Let's remedy that.
 
-Crucially, this inbound address must match the seed address we specified
-earlier in Alice and Bob's settings.
+`darkfi` provides a standard seed node called `lilith` that can act as
+the seed for many different protocols at the same time.
 
+Just like any p2p daemon, a seed node defines its networks settings
+from a config file, using the network type `Settings`. `lilith` allows
+for multiple networks to be configured in its config file. Crucially,
+each network must specify an `acccept_addr` which nodes on the network
+can connect to.

doc/src/learn/dchat/deployment/settings.md

@@ -6,79 +6,56 @@ will need to configure them using `net` type called
 This type consists of several settings that allow you to configure nodes
 in different ways.
 
-You would usually configure `Settings` using a config file or command
-line inputs. On dchat we are keeping things ultra simple. We pass a
-command line flag that is either `a` or `b`. If we pass `a` we will
-initialize the `Settings` for an inbound node. If we pass `b` we will
-initialize an outbound node.
+To do this, we'll create a default `dchatd_config.toml` file at the
+place specified in `CONFIG_FILE_CONTENTS`.
 
-Here's how that works. We define two methods called `alice()` and
-`bob()`. `alice()` returns the `Settings` that will create an inbound
-node. bob() return the `Settings` for an outbound node.
+```toml
+# dchatd toml
 
-We also implement logging that outputs to `/tmp/alice.log` and `/tmp/bob.log`
-so we can access the debug output of our nodes. We store this info in a
-log file because we don't want it interfering with our terminal UI when
-we eventually build it.
+[net]
+## P2P accept addresses Required for inbound nodes.
+inbound=["tcp://127.0.0.1:51554"]
 
-This is a function that returns the settings to create Alice, an
-inbound node:
+## P2P external addresses. Required for inbound nodes.
+external_addr=["tcp://127.0.0.1:51554"]
 
-```rust
-fn alice() -> Result<Settings> {
-   let log_level = simplelog::LevelFilter::Debug;
-   let log_config = simplelog::Config::default();
-
-   let log_path = "/tmp/alice.log";
-   let file = File::create(log_path).unwrap();
-   WriteLogger::init(log_level, log_config, file)?;
-
-   let seed = Url::parse("tcp://127.0.0.1:55555").unwrap();
-   let inbound = Url::parse("tcp://127.0.0.1:55554").unwrap();
-   let ext_addr = Url::parse("tcp://127.0.0.1:55554").unwrap();
+## Seed nodes to connect to. Required for inbound and outbound nodes.
+seeds=["tcp://127.0.0.1:50515"]
 
-   let settings = Settings {
-       inbound: Some(inbound),
-       external_addr: Some(ext_addr),
-       seeds: vec![seed],
-       ..Default::default()
-   };
+## Outbound connect slots. Required for outbound nodes.
+outbound_connections = 5
 
-   Ok(settings)
-}
 ```
 
-This is a function that returns the settings to create Bob, an
-outbound node:
-
-```rust
-fn bob() -> Result<Settings> {
-   let log_level = simplelog::LevelFilter::Debug;
-   let log_config = simplelog::Config::default();
-
-   let log_path = "/tmp/bob.log";
-   let file = File::create(log_path).unwrap();
-   WriteLogger::init(log_level, log_config, file)?;
-
-   let seed = Url::parse("tcp://127.0.0.1:55555").unwrap();
+Inbound nodes specify an external address and an inbound address: this is
+where it will receive connections. Outbound nodes specify the number of
+outbound connection slots, which is the number of outbound connections
+the node will try to make, and seed addresses from which it can receive
+information about other nodes it can connect to. If all of these settings
+are enabled, the node is both inbound and outbound, i.e. a full node.
 
-   let settings = Settings {
-       inbound: None,
-       outbound_connections: 5,
-       seeds: vec![seed],
-       ..Default::default()
-   };
+Next, we add `SettingsOpt` to our `Args` struct. This will allow us to
+read the fields specified in TOML as the darkfi `net` type, `Settings`.
 
-   Ok(settings)
+```rust
+#[derive(Clone, Debug, Deserialize, StructOpt, StructOptToml)]
+#[serde(default)]
+#[structopt(name = "dchat", about = cli_desc!())]
+struct Args {
+    #[structopt(short, long)]
+    /// Configuration file to use
+    config: Option<String>,
+
+    #[structopt(short, long)]
+    /// Set log file to ouput into
+    log: Option<String>,
+
+    #[structopt(short, parse(from_occurrences))]
+    /// Increase verbosity (-vvv supported)
+    verbose: u8,
+
+    /// P2P network settings
+    #[structopt(flatten)]
+    net: SettingsOpt,
 }
 ```
-
-Both outbound and inbound nodes specify a seed address to connect to. The
-inbound node also specifies an external address and an inbound address:
-this is where it will receive connections. The outbound node specifies
-the number of outbound connection slots, which is the number of outbound
-connections the node will try to make.
-
-These are the only settings we need to think about. For the rest, we
-use the network defaults.
-

doc/src/learn/dchat/deployment/start-run-stop.md

@@ -1,14 +1,8 @@
-# Start-Run-Stop
+# Start-Stop
 
 Now that we have initialized the network settings we can create an
 instance of the p2p network.
 
-Add the following to `main()`:
-
-```rust
-    let p2p = net::P2p::new(settings?).await;
-```
-
 We will next create a `Dchat` struct that will store all the data required
 by dchat. For now, it will just hold a pointer to the p2p network.
 
@@ -18,81 +12,110 @@ struct Dchat {
 }
 
 impl Dchat {
-    fn new(p2p: net::P2pPtr) -> Self {
+    fn new(
+        p2p: net::P2pPtr,
+    ) -> Self {
         Self { p2p }
     }
 }
 ```
 
-Now let's add a `start()` function to the `Dchat` implementation. `start()`
-takes an executor and runs three p2p methods, `p2p::start()`, `p2p::run()`,
-and `p2p::stop()`.
+Let's build out our `realmain` function as follows:
 
 ```rust
-    async fn start(&mut self, ex: Arc<Executor<'_>>) -> Result<()> {
-        let ex2 = ex.clone();
+use log::info;
 
-        self.p2p.clone().start(ex.clone()).await?;
-        ex2.spawn(self.p2p.clone().run(ex.clone())).detach();
+async_daemonize!(realmain);
+async fn realmain(args: Args, ex: Arc<smol::Executor<'static>>) -> Result<()> {
+    let p2p = net::P2p::new(args.net.into(), ex.clone()).await;
+    info!("Starting P2P network");
+    p2p.clone().start().await?;
 
-        self.p2p.stop().await;
+    let (signals_handler, signals_task) = SignalHandler::new(ex)?;
+    signals_handler.wait_termination(signals_task).await?;
+    info!("Caught termination signal, cleaning up and exiting...");
 
-        Ok(())
-    }
+    info!("Stopping P2P network");
+    p2p.stop().await;
 
+    Ok(())
+}
 ```
+
+Here, we instantiate the p2p network using `P2p::new`. We then start it
+by calling `start`, and handle shutdown signals to safely shutdown the
+network using `P2p::stop`.
+
 Let's take a quick look at the underlying p2p methods we're using here.
 
 ## Start
 
-This is [start()](https://github.com/darkrenaissance/darkfi/blob/master/src/net/p2p.rs#L135):
+This is [start](https://github.com/darkrenaissance/darkfi/blob/master/src/net/p2p.rs#L126):
 
 ```rust
-{{#include ../../../../../src/net/p2p.rs:start}}
-```
-
-`start()` changes the `P2pState` to `P2pState::Start` and runs a [seed
-session](https://github.com/darkrenaissance/darkfi/blob/master/src/net/session/seed_session.rs).
-
-This loops through the seed addresses specified in our `Settings` and
-tries to connect to them. The seed session either connects successfully,
-fails with an error or times out.
-
-If a seed node connects successfully, it runs a version exchange protocol,
-stores the channel in the p2p list of channels, and disconnects, removing
-the channel from the channel list.
+/// Starts inbound, outbound, and manual sessions.
+pub async fn start(self: Arc<Self>) -> Result<()> {
+    debug!(target: "net::p2p::start()", "P2P::start() [BEGIN]");
+    info!(target: "net::p2p::start()", "[P2P] Starting P2P subsystem");
+
+    // First attempt any set manual connections
+    for peer in &self.settings.peers {
+        self.session_manual().connect(peer.clone()).await;
+    }
 
-## Run
+    // Start the inbound session
+    if let Err(err) = self.session_inbound().start().await {
+        error!(target: "net::p2p::start()", "Failed to start inbound session!: {}", err);
+        self.session_manual().stop().await;
+        return Err(err)
+    }
 
-This is [run()](https://github.com/darkrenaissance/darkfi/blob/master/src/net/p2p.rs#L163):
+    // Start the outbound session
+    self.session_outbound().start().await;
 
-```rust
-{{#include ../../../../../src/net/p2p.rs:run}}
+    info!(target: "net::p2p::start()", "[P2P] P2P subsystem started");
+    Ok(())
+}
 ```
 
-`run()` changes the P2pState to `P2pState::Run`. It then calls `start()`
-on manual, inbound and outbound sessions that are contained with the
-`P2p` struct. The outcome of `start()` will depend on how your node is
-configured. `start()` will try to run each kind of session, but if the
-configuration doesn't match attemping to start a session will simply
-return without doing anything. For example, if you are an outbound node,
-`inbound.start()` will return with the following message:
+`start` attempts to start an `Inbound`, `Manual` or `Outbound` session,
+which will succeed or fail depending on how your TOML is configured. For
+example, if you are an outbound node, `session_inbound.start()` will
+return with the following message:
 
 ```rust
 info!(target: "net", "Not configured for accepting incoming connections.");
 ```
 
-`run()` then waits for a stop signal and shuts down the sessions when it
-is received.
+The function calls in `start` trigger the following processes:
+
+`session_manual.connect()`: tries to connect to any `peer` addresses we
+have specified in the TOML, using a `Connector`.
+
+`session_inbound.start()`: starts an `Acceptor` on the inbound address
+specified in the TOML, then creates and registers a `Channel` on that
+address.
+
+`session_outbound.start()': tries to establish a connection using a
+`Connector` to the number of slots we have specified in the TOML field
+`outbound_connections`. For every `Slot`, `run` tries to find a valid
+address we can connect to through `PeerDiscovery`, which loops through
+all connected channels and sends out a `GetAddr` message. If we don't
+have any connected channels, `run` performs a `SeedSync`.
 
 ## Stop
 
-This is [stop()](https://github.com/darkrenaissance/darkfi/blob/master/src/net/p2p.rs#L306).
+This is [stop()](https://github.com/darkrenaissance/darkfi/blob/master/src/net/p2p.rs#L164).
 
 ```rust
-    {{#include ../../../../../src/net/p2p.rs:stop}}
+/// Stop the running P2P subsystem
+pub async fn stop(&self) {
+    // Stop the sessions
+    self.session_manual().stop().await;
+    self.session_inbound().stop().await;
+    self.session_outbound().stop().await;
+}
 ```
 
-`stop()` transmits a shutdown signal to all channels subscribed to the
+`stop` transmits a shutdown signal to all channels subscribed to the
 stop signal and safely shuts down the network.
-

doc/src/learn/dchat/deployment/writing-a-daemon.md

@@ -2,41 +2,58 @@
 
 DarkFi consists of many seperate daemons communicating with each other. To
 run the p2p network, we'll need to implement our own daemon.  So we'll
-start building `dchat` by configuring our main function into a daemon that
-can run the p2p network.
+start building `dchat` by creating a daemon that we call `dchatd`.
 
-```rust
-{{#include ../../../../../example/dchat/src/main.rs:daemon_deps}}
-
-#[async_std::main]
-async fn main() -> Result<()> {
-    let ex = Arc::new(Executor::new());
-    let ex2 = ex.clone();
+To do this, we'll make use of a DarkFi macro called
+[async_daemonize](https://github.com/darkrenaissance/darkfi/blob/0aba4e7864d459301a6c5afd8bda6a3d9f240b86/src/util/cli.rs).
 
-    let nthreads = std::thread::available_parallelism().unwrap().get();
-    let (signal, shutdown) = smol::channel::unbounded::<()>();
+`async_daemonize`is the standard way of daemonizing darkfi binaries. It
+implements TOML config file configuration, argument parsing and a
+multithreaded async executor that can be passed into the given function.
 
-    let (_, result) = Parallel::new()
-        .each(0..nthreads, |_| smol::future::block_on(ex2.run(shutdown.recv())))
-        .finish(|| {
-            smol::future::block_on(async move {
-                drop(signal);
-                Ok(())
-            })
-        });
+We use `async_daemonize` as follows:
 
-    result
+```rust
+use darkfi::{async_daemonize, cli_desc, Result};
+use smol::stream::StreamExt;
+use structopt_toml::{serde::Deserialize, structopt::StructOpt, StructOptToml};
+
+const CONFIG_FILE: &str = "dchatd_config.toml";
+const CONFIG_FILE_CONTENTS: &str = include_str!("../dchatd_config.toml");
+
+#[derive(Clone, Debug, Deserialize, StructOpt, StructOptToml)]
+#[serde(default)]
+#[structopt(name = "daemond", about = cli_desc!())]
+struct Args {
+    #[structopt(short, long)]
+    /// Configuration file to use
+    config: Option<String>,
+
+    #[structopt(short, long)]
+    /// Set log file to ouput into
+    log: Option<String>,
+
+    #[structopt(short, parse(from_occurrences))]
+    /// Increase verbosity (-vvv supported)
+    verbose: u8,
+}
 
+async_daemonize!(realmain);
+async fn realmain(args: Args, ex: Arc<smol::Executor<'static>>) -> Result<()> {
+    println!("Hello, world!");
+    Ok(())
 }
 ```
 
-We get the number of cpu cores using
-`std::thread::available_parallelism()` and spin up a bunch of threads
-in parallel using `easy_parallel`. Right now it doesn't do anything,
-but soon we'll run dchat inside this block.
+Behind the scenes, `async_daemonize` uses `structopt` and `structopt_toml`
+crates to build command line arguments as a struct called `Args`. It spins
+up a async executor using parallel threads, and implements signal handling
+to properly terminate the daemon on receipt of a stop signal.
 
-**Note**: DarkFi includes a macro called `async_daemonize` that is used by
-DarkFi binaries to minimize boilerplate in the repo.  To keep things
-simple we will ignore this macro for the purpose of this tutorial.  But
-check it out if you are curious: [util/cli.rs](https://github.com/darkrenaissance/darkfi/blob/master/src/util/cli.rs#L154).
+`async_daemonize` allow us to spawn the config data we specify at
+`CONFIG_FILE_CONTENTS` into a directory either specified using the
+command-line flag `--config`, or in the default darkfi config directory.
 
+`async_daemonize` also implements logging that will output
+different levels of debug info to the terminal, or to both the terminal
+and a log file if a log file is specified.