1. How to run and use the Trace Route on Mac OS? Trace Route is a utility that records and displays the route through the Internet from your computer to a specified destination computer or server. It also counts and reveals the amount of time spent during each 'hop' (the route taken by a data packet from one node to another in the network).
  2. Mac App Store is the simplest way to find and download apps for your Mac. To download apps from the Mac App Store, you need a Mac with OS X 10.6.6 or later.
  1. Dirt Race Mac Os Download
  2. Dirt Race Mac Os Update
  3. Dirt Race Mac Os 7
  4. Dirt Race Mac Os X

Dirt Race Mac Os Download

I wanted to trace the system calls made by the find command to debug some performance issues however I could not figure out how to do this on Mac OS X Yosemite. How can I trace system calls for an arbitrary program similarly to what strace does on FreeBSD? I am especially interested in tracing file-system related calls. Non kernel components of DTrace on Mac OS. Contribute to opendtrace/macos-dtrace development by creating an account on GitHub.

A few days ago, I was stymied at work by a set of tests that had intermittentfailures on OSX but not Windows.

There was a process which would try to obtain an exclusive lock on a file,using the lock-on-open provided by the BSD/MacOS O_EXLOCK flag to open(2).It also used O_NONBLOCK; if the file was locked by another process, it couldbe skipped. The process would hold the lock and remove(unlink(2)) the file,beforeclose(2)-ing the descriptor. My understanding was that only one process wouldbe able to access the file, ever. The test harness did not agree. It was clearthat if multiple processes managed to race on the same file, they could bothend up with valid file descriptors. I should clarify that just realizing thistook a lot of print statements over several hours over 2 days. It is unusual torun into unexpected behavior from core system calls, which made me doubt myselfa lot.

DirtRace Mac OS

It was easy to fix the test harness; we had unique IDs in the file and we couldjust de-duplicate across that. However, the problem kept haunting me (it wasmaking me question my understanding of operating systems!), and so I keptdigging around at home.

The first task was to write an extremely simple program that would reproducethe problem. Interacting with POSIX is really easy in Rust and absolves you ofhaving to do C string manipulations, so that is what I used.

Reproducing the problem

(All code was run on OSX 10.12.5 with HFS+.)

The full code is online,including the sample output included here and the DTrace script. Here are thecore file operations.

We try to open the file with the requisite flags, and stop if it fails. Wesleep for a bit to simulate the work the original program was doing, and alsobecause it helps make the problem easier to reproduce. We try to remove thefile. Finally, we rely on std::fs::File’s destructor close(2)ing the filedescriptor when file goes out of scope.

Running two instances of this program on some sample files would sometimeslead to output like:

Mac

The first column on each line is the Process ID. This didn’t make any sense.79011 was opening the file successfully, which meant it must be locked. Itsunlink wasn’t failing, and yet, 79012 had managed to successfully open a filethat had been deleted (as evidenced by the inability to remove it). If the openwas succeeding, that meant 79011 had already released the lock, so it must haveremoved the file. If you are following along with the code, you may need to runthe program as instructed in the README, several times, to see the issuehappen.

Standard print statements and debuggers are useless to debug this. With twodifferent processes, we can’t rely on the individual logs to be ordered, whichmeans we can’t reason precisely. Since debuggers can’t cross system callboundaries, we can’t use them to peek under the hood. Plus, a debugger maythrow off the timing and not cause the problem to reproduce. Enter DTrace.

DTrace is a tracing framework originally designed for Solaris, that allowsinspecting system internals at a very low cost, and is zero cost when disabled.It was ported to MacOS a few years ago. I’ve been aware of DTrace for a fewyears, but never had the opportunity to use it till now.

I’m no DTrace expert, so I first turned to the scripts OSX ships with (whichare all based onwork by Brendan Gregg), to see if there was something thatwould do the job. Turns out there is a wrapper called opensnoop that tracksopen(2) calls, including filtering by PID or process names. We want to tracknot just open(2), but also close(2) and unlink(2). The modified script isuncreatively called openclosesnoop1. Here is the sample output from the aboverun (You’ve to launch the snoop before you run the program).

We see that our intuitions are at least correct in terms of syscall ordering.We also have a reliable ordering of events, and DTrace shows us the argumentsan results of the system calls. All these lines are printed when the systemcall returns. The output columns are:

  1. system call name
  2. a timestamp maintained by DTrace in microseconds since some arbitrary pointin the past. It should only be used for relative ordering.
  3. Wall time.
  4. UID
  5. PID
  6. Return value of the system call.
  7. Flags passed to system call. 0 for unlink and close.
  8. errno. 0 indicates success.
  9. Path for open and unlink, file descriptor for close.
  10. Program name

We see that the open calls succeed and lead to file descriptor 3 for bothprograms. The 79011 unlink succeeds, the 79012 fails. Both are able to open andclose the file. Clearly the locking isn’t being violated, nor is unlink; thesecond call fails. What else could be wrong? Let’s try tracking not just whensystem calls return, but when they are entered. Here is the output from anotherrun, with only open being tracked for entry, since that is the one messing upour assumptions.

That is very interesting! 75939 enters open pretty soon after 75938 hassuccessfully opened 7.sample. Then, 75939 is scheduled off the CPU, 75938’sunlink and close calls are scheduled and run to completion. 75939’s open thencompletes successfully! Its attempt to unlink fails, since the file has beenunlinked and can no longer be accessed by file path.

Dirt Race Mac Os Update

What’s going on?

Now that we know we enter open(2) then get booted off CPU, let’s go back andread the man page.

Dirt Race Mac Os 7

open(2) is acquiring a file descriptor with no locks. It manages to do thisbefore 75938’s unlink has occurred. This ensures that 75939 now has the fileopen, so unlink will not reclaim resources, but the link count will getdecremented to zero and the file no longer accessible by path. When controlreturns to this process (or rather to the system call running on its behalf),the file’s lock (remember the file is independent of link count, usuallyidentified by an inode or similar) has been released and the flock(2) portionfinishes successfully.

That is, open(O_EXLOCK O_NONBLOCK O_RDONLY) maps roughly to:

Dirt Race Mac Os X

and behaves just like the above would if it ran in userspace. The kernelprovides no atomicity on the path name.

The root cause was a failure to read the fine print, and filling in assumptionswhere things were left unsaid. POSIX does not require open(2) to be atomic inthe general case. Certain pathnameoperationsare atomic, but only the open(path, O_CREAT O_EXCL, ...) operation fallsinto that category.

This was a really fun puzzle to solve. It is remarkable how easy it is to debugissues when you can see exactly what the kernel is doing, with great precisionand fidelity. I’ve mainly used DTrace for syscall tracing, but it is capable ofa lot more, including inspecting I/O events, mutex/lock states and processscheduling. MacOS ships with several out-of-the-box scripts that can be listedwith man -k dtrace. I highly encourage getting familiar with the tool. If youare on Linux, eBPF is a similar alternative.

  1. It took me a while to realize close(2) maps to close_nocancel in DTrace. [return]

I was doing some work on my Python bindings for libusdt which reminded me about how annoying it is when DTrace isn't working.

The solution (as I noted here) is pretty simple:

  1. Reboot into Recovery Mode (hold down ⌘R during boot)
  2. Launch a shell and run csrutil enable --without dtrace
  3. Reboot and allow machine to boot normally

Embarrassing story time:
This is neatly documented by Brian Bennett in a StackExchange post (from 2015!). Not only that, but I even left myself a breadcrumb linking to that StackExchange post in May of 2016. But sadly none of that stopped me from taking to Twitter to ask how to solve this problem.

Oh well. At least now I can successfully run the test suites for libusdt and my Python wrappers for it locally on my laptop.

⇐ ⇐ Planetary Sanitations Inc. Mac OS
⇒ ⇒ Key Slaughter Mac OS