SystemVerilog Fork Disable "Gotchas"

This is a long post with a lot of SystemVerilog code. The purpose of this entry is to hopefully save you from beating your head against the wall trying to figure out some of the subtleties of SystemVerilog processes (basically, threads). Subtleties like these are commonly referred to in the industry as "Gotchas" which makes them sound so playful and fun, but they really aren't either.

I encourage you to run these examples with your simulator (if you have access to one) so that a) you can see the results first hand and better internalize what's going on, and b) you can tell me in the comments if this code works fine for you and I'll know I should go complain to my simulator vendor.

OK, I'll start with a warm-up that everyone who writes any Verilog or SystemVerilog at all should be aware of, tasks are static by default. If you do this:

module top;
   task do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork 
         do_stuff(10);
         do_stuff(5);
      join 
   end 
endmodule 

both do_stuff calls will wait for 5 time units, and you see this:

waited 5, then did stuff
waited 5, then did stuff

I suppose being static by default is a performance/memory-use optimization, but it's guaranteed to trip up programmers who started with different languages. The fix is to make the task "automatic" instead of static:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork 
         do_stuff(10);
         do_stuff(5);
      join 
   end 
endmodule 

And now you get what you expected:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork 
         do_stuff(10);
         do_stuff(5);
      join_any 
      $display("fork has been joined");
   end 
endmodule 

You'll get this output:

waited 5, then did stuff
fork has been joined
waited 10, then did stuff

That's fine, but that extra action from the slower do_stuff after the fork-join_any block has finished might not be what you wanted. You can name the fork block and disable it to take care of that, like so:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork : do_stuff_fork
         do_stuff(10);
         do_stuff(5);
      join_any 
      $display("fork has been joined");
      disable do_stuff_fork;
   end 
endmodule 

Unless your simulator, like mine, "in the current release" will not disable sub-processes created by a fork-join_any statement. Bummer. It's OK, though, because SystemVerilog provides a disable fork statement that disables all active threads of a calling process (if that description doesn't already make you nervous, just wait). Simply do this:

module top;
   task automatic do_stuff(int wait_time);
      #wait_time $display("waited %0d, then did stuff", wait_time);
   endtask 
 
   initial begin 
      fork : do_stuff_fork
         do_stuff(10);
         do_stuff(5);
      join_any 
      $display("fork has been joined");
      disable fork;
   end 
endmodule 

And you get:

waited 5, then did stuff
fork has been joined

Nothing wrong there. Now let's say you have a class that is monitoring a bus. Using a classes are cool because if you have two buses you can create two instances of your monitor class, one for each bus. We can expand our code example to approximate this scenario, like so:

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask 
 
   task monitor();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable do_stuff_fork;
   endtask 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join 
      $display("main fork has been joined");
   end 
endmodule 

Note that I went back to disabling the fork by name instead of using the disable fork statement. This is to illustrate another gotcha. That disable call will disable both instances of the fork, monitor 1's instance and monitor 2's. You get this output:

monitor 1 waited 6, then did stuff
monitor 1 fork has been joined
monitor 2 fork has been joined
main fork has been joined

Because disabling by name is such a blunt instrument, poor monitor 2 never got a chance. Now, if you turn the disable into a disable fork, like so:

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask 
 
   task monitor();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
   endtask 
 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join 
      $display("main fork has been joined");
   end 
endmodule 

You get what you expect:

monitor 1 waited 6, then did stuff
monitor 1 fork has been joined
monitor 2 waited 7, then did stuff
monitor 2 fork has been joined
main fork has been joined

It turns out that, like when you disable something by name, disable fork is a pretty blunt tool also. Remember my ominous parenthetical "just wait" above? Here it comes. Try adding another fork like this (look for the fork_something task call):

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   function void fork_something();
      fork 
         # 300 $display("monitor %0d: you'll never see this", id);
      join_none 
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask 
 
   task monitor();
      fork_something();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
   endtask 
 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
 
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join 
      $display("main fork has been joined");
   end 
endmodule

The output you get is:

monitor 1 waited 6, then did stuff
monitor 1 fork has been joined
monitor 2 waited 7, then did stuff
monitor 2 fork has been joined
main fork has been joined

Yup, fork_something's fork got disabled too. How do you disable only the processes inside the fork you want? You have to wrap your fork-join_any inside of a fork-join, of course. That makes sure that there aren't any other peers or child processes for disable fork to hit. Here's the zoomed in view of that (UPDATE: added missing begin...end for outer fork):

task monitor();
   fork_something();
   fork begin 
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
   end 
   join 
endtask 

And now you get what you expect:

monitor 2 fork has been joined
monitor 1 fork has been joined
monitor 1 waited 6, then did stuff
monitor 2 waited 7, then did stuff
main fork has been joined
monitor 1 waited 11, then did stuff
monitor 2 waited 12, then did stuff
monitor 2: you'll never see this
monitor 1: you'll never see this

So, wrap your fork-join_any inside a fork-join or else it's, "Gotcha!!!" (I can almost picture the SystemVerilog language designers saying that out loud, with maniacal expressions on their faces).

But wait, I discovered something even weirder. Instead of making that wrapper fork, you can just move the fork_something() call after the disable fork call and then it doesn't get disabled (you actually see the "you'll never see this" message, try it). So, you might think, just reordering your fork and disable fork calls and that will fix your problem. It will, unless (I learned by sad experience) the monitor task is being repeatedly called inside a forever loop. Here's a simplification of the code that really inspired me to write this all up:

class a_bus_monitor;
   int id;
 
   function new(int id_in);
      id = id_in;
   endfunction 
 
   function void fork_something();
      fork 
         # 30 $display("monitor %0d: you'll never see this", id);
      join_none 
   endfunction 
 
   task automatic do_stuff(int wait_time);
      #wait_time $display("monitor %0d waited %0d, then did stuff", id, wait_time);
   endtask // do_stuff 
 
   task monitor_subtask();
      fork : do_stuff_fork
         do_stuff(10 + id);
         do_stuff(5 + id);
      join_any 
      $display("monitor %0d fork has been joined", id);
      disable fork;
      fork_something();
   endtask 
 
   task monitor();
      forever begin 
         monitor_subtask();
      end 
   endtask 
 
endclass 
 
module top;
   a_bus_monitor abm1;
   a_bus_monitor abm2;
 
   initial begin 
      abm1 = new(1);
      abm2 = new(2);
      fork 
         abm2.monitor();
         abm1.monitor();
      join_none 
      $display("main fork has been joined");
      # 60 $finish;
   end 
endmodule 

The fork inside the fork_something task will get disabled before it can do its job, even though it's after the disable fork statement.

My advice? Just always wrap any disable fork calls inside a fork-join.