Intel(R) Threading Building Blocks Doxygen Documentation  version 4.2.3
tbb::internal::arena Class Reference

#include <arena.h>

Inheritance diagram for tbb::internal::arena:
Collaboration diagram for tbb::internal::arena:

Public Types

enum  new_work_type { work_spawned, wakeup, work_enqueued }
 Types of work advertised by advertise_new_work() More...
 
typedef padded< arena_basebase_type
 
typedef uintptr_t pool_state_t
 

Public Member Functions

 arena (market &, unsigned max_num_workers, unsigned num_reserved_slots)
 Constructor. More...
 
mail_outboxmailbox (affinity_id id)
 Get reference to mailbox corresponding to given affinity_id. More...
 
void free_arena ()
 Completes arena shutdown, destructs and deallocates it. More...
 
unsigned num_workers_active () const
 The number of workers active in the arena. More...
 
bool is_recall_requested () const
 Check if the recall is requested by the market. More...
 
template<arena::new_work_type work_type>
void advertise_new_work ()
 If necessary, raise a flag that there is new job in arena. More...
 
bool is_out_of_work ()
 Check if there is job anywhere in arena. More...
 
void enqueue_task (task &, intptr_t, FastRandom &)
 enqueue a task into starvation-resistance queue More...
 
void process (generic_scheduler &)
 Registers the worker with the arena and enters TBB scheduler dispatch loop. More...
 
template<unsigned ref_param>
void on_thread_leaving ()
 Notification that worker or master leaves its arena. More...
 
bool has_enqueued_tasks ()
 Check for the presence of enqueued tasks at all priority levels. More...
 
template<bool as_worker>
size_t occupy_free_slot (generic_scheduler &s)
 Tries to occupy a slot in the arena. On success, returns the slot index; if no slot is available, returns out_of_arena. More...
 
size_t occupy_free_slot_in_range (generic_scheduler &s, size_t lower, size_t upper)
 Tries to occupy a slot in the specified range. More...
 

Static Public Member Functions

static arenaallocate_arena (market &, unsigned num_slots, unsigned num_reserved_slots)
 Allocate an instance of arena. More...
 
static int unsigned num_arena_slots (unsigned num_slots)
 
static int allocation_size (unsigned num_slots)
 
static bool is_busy_or_empty (pool_state_t s)
 No tasks to steal or snapshot is being taken. More...
 

Public Attributes

arena_slot my_slots [1]
 
- Public Attributes inherited from tbb::internal::padded_base< arena_base, NFS_MaxLineSize, sizeof(arena_base) % NFS_MaxLineSize >
char pad [S - R]
 
- Public Attributes inherited from tbb::internal::arena_base
unsigned my_num_workers_allotted
 The number of workers that have been marked out by the resource manager to service the arena. More...
 
atomic< unsigned > my_references
 Reference counter for the arena. More...
 
atomic< unsigned > my_limit
 The maximal number of currently busy slots. More...
 
task_stream< num_priority_levelsmy_task_stream
 Task pool for the tasks scheduled via task::enqueue() method. More...
 
unsigned my_max_num_workers
 The number of workers requested by the master thread owning the arena. More...
 
int my_num_workers_requested
 The number of workers that are currently requested from the resource manager. More...
 
tbb::atomic< uintptr_t > my_pool_state
 Current task pool state and estimate of available tasks amount. More...
 
marketmy_market
 The market that owns this arena. More...
 
uintptr_t my_aba_epoch
 ABA prevention marker. More...
 
cpu_ctl_env my_cpu_ctl_env
 FPU control settings of arena's master thread captured at the moment of arena instantiation. More...
 
unsigned my_num_slots
 The number of slots in the arena. More...
 
unsigned my_num_reserved_slots
 The number of reserved slots (can be occupied only by masters). More...
 
concurrent_monitor my_exit_monitors
 Waiting object for master threads that cannot join the arena. More...
 
- Public Attributes inherited from tbb::internal::padded_base< intrusive_list_node, NFS_MaxLineSize, sizeof(intrusive_list_node) % NFS_MaxLineSize >
char pad [S - R]
 
- Public Attributes inherited from tbb::internal::intrusive_list_node
intrusive_list_nodemy_prev_node
 
intrusive_list_nodemy_next_node
 

Static Public Attributes

static const pool_state_t SNAPSHOT_EMPTY = 0
 No tasks to steal since last snapshot was taken. More...
 
static const pool_state_t SNAPSHOT_FULL = pool_state_t(-1)
 At least one task has been offered for stealing since the last snapshot started. More...
 
static const unsigned ref_external_bits = 12
 The number of least significant bits for external references. More...
 
static const unsigned ref_external = 1
 Reference increment values for externals and workers. More...
 
static const unsigned ref_worker = 1<<ref_external_bits
 
static const size_t out_of_arena = ~size_t(0)
 

Private Member Functions

void restore_priority_if_need ()
 If enqueued tasks found, restore arena priority and task presence status. More...
 

Detailed Description

Definition at line 272 of file arena.h.

Member Typedef Documentation

◆ base_type

Definition at line 277 of file arena.h.

◆ pool_state_t

Definition at line 311 of file arena.h.

Member Enumeration Documentation

◆ new_work_type

Types of work advertised by advertise_new_work()

Enumerator
work_spawned 
wakeup 
work_enqueued 

Definition at line 280 of file arena.h.

Constructor & Destructor Documentation

◆ arena()

tbb::internal::arena::arena ( market m,
unsigned  max_num_workers,
unsigned  num_reserved_slots 
)

Constructor.

Definition at line 182 of file arena.cpp.

182  {
183  __TBB_ASSERT( !my_guard, "improperly allocated arena?" );
184  __TBB_ASSERT( sizeof(my_slots[0]) % NFS_GetLineSize()==0, "arena::slot size not multiple of cache line size" );
185  __TBB_ASSERT( (uintptr_t)this % NFS_GetLineSize()==0, "arena misaligned" );
186 #if __TBB_TASK_PRIORITY
187  __TBB_ASSERT( !my_reload_epoch && !my_orphaned_tasks && !my_skipped_fifo_priority, "New arena object is not zeroed" );
188 #endif /* __TBB_TASK_PRIORITY */
189  my_market = &m;
190  my_limit = 1;
191  // Two slots are mandatory: for the master, and for 1 worker (required to support starvation resistant tasks).
192  my_num_slots = num_arena_slots(num_slots);
193  my_num_reserved_slots = num_reserved_slots;
194  my_max_num_workers = num_slots-num_reserved_slots;
195  my_references = ref_external; // accounts for the master
196 #if __TBB_TASK_PRIORITY
197  my_bottom_priority = my_top_priority = normalized_normal_priority;
198 #endif /* __TBB_TASK_PRIORITY */
199  my_aba_epoch = m.my_arenas_aba_epoch;
200 #if __TBB_ARENA_OBSERVER
201  my_observers.my_arena = this;
202 #endif
203 #if __TBB_PREVIEW_RESUMABLE_TASKS
204  my_co_cache.init(4 * num_slots);
205 #endif
207  // Construct slots. Mark internal synchronization elements for the tools.
208  for( unsigned i = 0; i < my_num_slots; ++i ) {
209  __TBB_ASSERT( !my_slots[i].my_scheduler && !my_slots[i].task_pool, NULL );
210  __TBB_ASSERT( !my_slots[i].task_pool_ptr, NULL );
211  __TBB_ASSERT( !my_slots[i].my_task_pool_size, NULL );
212 #if __TBB_PREVIEW_RESUMABLE_TASKS
213  __TBB_ASSERT( !my_slots[i].my_scheduler_is_recalled, NULL );
214 #endif
215  ITT_SYNC_CREATE(my_slots + i, SyncType_Scheduler, SyncObj_WorkerTaskPool);
216  mailbox(i+1).construct();
217  ITT_SYNC_CREATE(&mailbox(i+1), SyncType_Scheduler, SyncObj_Mailbox);
218  my_slots[i].hint_for_pop = i;
219 #if __TBB_PREVIEW_CRITICAL_TASKS
220  my_slots[i].hint_for_critical = i;
221 #endif
222 #if __TBB_STATISTICS
223  my_slots[i].my_counters = new ( NFS_Allocate(1, sizeof(statistics_counters), NULL) ) statistics_counters;
224 #endif /* __TBB_STATISTICS */
225  }
227  ITT_SYNC_CREATE(&my_task_stream, SyncType_Scheduler, SyncObj_TaskStream);
228 #if __TBB_PREVIEW_CRITICAL_TASKS
229  my_critical_task_stream.initialize(my_num_slots);
230  ITT_SYNC_CREATE(&my_critical_task_stream, SyncType_Scheduler, SyncObj_CriticalTaskStream);
231 #endif
232 #if __TBB_ENQUEUE_ENFORCED_CONCURRENCY
233  my_local_concurrency_mode = false;
234  my_global_concurrency_mode = false;
235 #endif
236 #if !__TBB_FP_CONTEXT
238 #endif
239 }
static const unsigned ref_external
Reference increment values for externals and workers.
Definition: arena.h:323
task_stream< num_priority_levels > my_task_stream
Task pool for the tasks scheduled via task::enqueue() method.
Definition: arena.h:168
uintptr_t my_aba_epoch
ABA prevention marker.
Definition: arena.h:231
unsigned my_num_slots
The number of slots in the arena.
Definition: arena.h:246
cpu_ctl_env my_cpu_ctl_env
FPU control settings of arena's master thread captured at the moment of arena instantiation.
Definition: arena.h:235
void *__TBB_EXPORTED_FUNC NFS_Allocate(size_t n_element, size_t element_size, void *hint)
Allocate memory on cache/sector line boundary.
void construct()
Construct *this as a mailbox from zeroed memory.
Definition: mailbox.h:158
unsigned my_max_num_workers
The number of workers requested by the master thread owning the arena.
Definition: arena.h:181
void initialize(unsigned n_lanes)
Definition: task_stream.h:83
atomic< unsigned > my_references
Reference counter for the arena.
Definition: arena.h:149
unsigned hint_for_pop
Hint provided for operations with the container of starvation-resistant tasks.
static int unsigned num_arena_slots(unsigned num_slots)
Definition: arena.h:292
arena_slot my_slots[1]
Definition: arena.h:386
unsigned my_num_reserved_slots
The number of reserved slots (can be occupied only by masters).
Definition: arena.h:249
market * my_market
The market that owns this arena.
Definition: arena.h:228
#define ITT_SYNC_CREATE(obj, type, name)
Definition: itt_notify.h:119
mail_outbox & mailbox(affinity_id id)
Get reference to mailbox corresponding to given affinity_id.
Definition: arena.h:301
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
atomic< unsigned > my_limit
The maximal number of currently busy slots.
Definition: arena.h:157
size_t __TBB_EXPORTED_FUNC NFS_GetLineSize()
Cache/sector line size.

References __TBB_ASSERT, tbb::internal::mail_outbox::construct(), tbb::internal::cpu_ctl_env::get_env(), tbb::internal::arena_slot_line2::hint_for_pop, tbb::internal::task_stream< Levels >::initialize(), ITT_SYNC_CREATE, mailbox(), tbb::internal::arena_base::my_aba_epoch, tbb::internal::market::my_arenas_aba_epoch, tbb::internal::arena_base::my_cpu_ctl_env, tbb::internal::arena_base::my_limit, tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_max_num_workers, tbb::internal::arena_base::my_num_reserved_slots, tbb::internal::arena_base::my_num_slots, tbb::internal::arena_base::my_references, my_slots, tbb::internal::arena_base::my_task_stream, tbb::internal::NFS_Allocate(), tbb::internal::NFS_GetLineSize(), num_arena_slots(), and ref_external.

Referenced by allocate_arena().

Here is the call graph for this function:
Here is the caller graph for this function:

Member Function Documentation

◆ advertise_new_work()

template<arena::new_work_type work_type>
void tbb::internal::arena::advertise_new_work ( )

If necessary, raise a flag that there is new job in arena.

Definition at line 480 of file arena.h.

480  {
481  if( work_type == work_enqueued ) {
482 #if __TBB_ENQUEUE_ENFORCED_CONCURRENCY
483  if ( as_atomic(my_market->my_num_workers_soft_limit) == 0 && as_atomic(my_global_concurrency_mode) == false )
484  my_market->enable_mandatory_concurrency(this);
485 
486  if ( my_max_num_workers == 0 && my_num_reserved_slots == 1 ) {
487  __TBB_ASSERT(!my_local_concurrency_mode, NULL);
488  my_local_concurrency_mode = true;
490  my_max_num_workers = 1;
492  return;
493  }
494 #endif /* __TBB_ENQUEUE_ENFORCED_CONCURRENCY */
495  // Local memory fence here and below is required to avoid missed wakeups; see the comment below.
496  // Starvation resistant tasks require concurrency, so missed wakeups are unacceptable.
497  atomic_fence();
498  }
499  else if( work_type == wakeup ) {
500  __TBB_ASSERT(my_max_num_workers!=0, "Unexpected worker wakeup request");
501  atomic_fence();
502  }
503  // Double-check idiom that, in case of spawning, is deliberately sloppy about memory fences.
504  // Technically, to avoid missed wakeups, there should be a full memory fence between the point we
505  // released the task pool (i.e. spawned task) and read the arena's state. However, adding such a
506  // fence might hurt overall performance more than it helps, because the fence would be executed
507  // on every task pool release, even when stealing does not occur. Since TBB allows parallelism,
508  // but never promises parallelism, the missed wakeup is not a correctness problem.
509  pool_state_t snapshot = my_pool_state;
510  if( is_busy_or_empty(snapshot) ) {
511  // Attempt to mark as full. The compare_and_swap below is a little unusual because the
512  // result is compared to a value that can be different than the comparand argument.
513  if( my_pool_state.compare_and_swap( SNAPSHOT_FULL, snapshot )==SNAPSHOT_EMPTY ) {
514  if( snapshot!=SNAPSHOT_EMPTY ) {
515  // This thread read "busy" into snapshot, and then another thread transitioned
516  // my_pool_state to "empty" in the meantime, which caused the compare_and_swap above
517  // to fail. Attempt to transition my_pool_state from "empty" to "full".
518  if( my_pool_state.compare_and_swap( SNAPSHOT_FULL, SNAPSHOT_EMPTY )!=SNAPSHOT_EMPTY ) {
519  // Some other thread transitioned my_pool_state from "empty", and hence became
520  // responsible for waking up workers.
521  return;
522  }
523  }
524  // This thread transitioned pool from empty to full state, and thus is responsible for
525  // telling the market that there is work to do.
526 #if __TBB_ENQUEUE_ENFORCED_CONCURRENCY
527  if( work_type == work_spawned ) {
528  if( my_local_concurrency_mode ) {
530  __TBB_ASSERT(!governor::local_scheduler()->is_worker(), "");
531  // There was deliberate oversubscription on 1 core for sake of starvation-resistant tasks.
532  // Now a single active thread (must be the master) supposedly starts a new parallel region
533  // with relaxed sequential semantics, and oversubscription should be avoided.
534  // Demand for workers has been decreased to 0 during SNAPSHOT_EMPTY, so just keep it.
535  my_max_num_workers = 0;
536  my_local_concurrency_mode = false;
537  return;
538  }
539  if ( as_atomic(my_global_concurrency_mode) == true )
540  my_market->mandatory_concurrency_disable( this );
541  }
542 #endif /* __TBB_ENQUEUE_ENFORCED_CONCURRENCY */
543  // TODO: investigate adjusting of arena's demand by a single worker.
545  }
546  }
547 }
static const pool_state_t SNAPSHOT_EMPTY
No tasks to steal since last snapshot was taken.
Definition: arena.h:314
tbb::atomic< uintptr_t > my_pool_state
Current task pool state and estimate of available tasks amount.
Definition: arena.h:191
unsigned my_num_workers_soft_limit
Current application-imposed limit on the number of workers (see set_active_num_workers())
Definition: market.h:78
atomic< T > & as_atomic(T &t)
Definition: atomic.h:572
uintptr_t pool_state_t
Definition: arena.h:311
static const pool_state_t SNAPSHOT_FULL
At least one task has been offered for stealing since the last snapshot started.
Definition: arena.h:317
unsigned my_max_num_workers
The number of workers requested by the master thread owning the arena.
Definition: arena.h:181
void atomic_fence()
Sequentially consistent full memory fence.
Definition: tbb_machine.h:339
void adjust_demand(arena &, int delta)
Request that arena's need in workers should be adjusted.
Definition: market.cpp:556
static generic_scheduler * local_scheduler()
Obtain the thread-local instance of the TBB scheduler.
Definition: governor.h:129
unsigned my_num_reserved_slots
The number of reserved slots (can be occupied only by masters).
Definition: arena.h:249
market * my_market
The market that owns this arena.
Definition: arena.h:228
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
static bool is_busy_or_empty(pool_state_t s)
No tasks to steal or snapshot is being taken.
Definition: arena.h:327

References __TBB_ASSERT, tbb::internal::market::adjust_demand(), tbb::internal::as_atomic(), tbb::atomic_fence(), is_busy_or_empty(), tbb::internal::governor::local_scheduler(), tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_max_num_workers, tbb::internal::arena_base::my_num_reserved_slots, tbb::internal::market::my_num_workers_soft_limit, tbb::internal::arena_base::my_pool_state, SNAPSHOT_EMPTY, SNAPSHOT_FULL, wakeup, work_enqueued, and work_spawned.

Referenced by tbb::internal::generic_scheduler::get_task(), tbb::internal::generic_scheduler::local_spawn(), and tbb::internal::generic_scheduler::steal_task_from().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ allocate_arena()

arena & tbb::internal::arena::allocate_arena ( market m,
unsigned  num_slots,
unsigned  num_reserved_slots 
)
static

Allocate an instance of arena.

Definition at line 241 of file arena.cpp.

241  {
242  __TBB_ASSERT( sizeof(base_type) + sizeof(arena_slot) == sizeof(arena), "All arena data fields must go to arena_base" );
243  __TBB_ASSERT( sizeof(base_type) % NFS_GetLineSize() == 0, "arena slots area misaligned: wrong padding" );
244  __TBB_ASSERT( sizeof(mail_outbox) == NFS_MaxLineSize, "Mailbox padding is wrong" );
245  size_t n = allocation_size(num_arena_slots(num_slots));
246  unsigned char* storage = (unsigned char*)NFS_Allocate( 1, n, NULL );
247  // Zero all slots to indicate that they are empty
248  memset( storage, 0, n );
249  return *new( storage + num_arena_slots(num_slots) * sizeof(mail_outbox) ) arena(m, num_slots, num_reserved_slots);
250 }
padded< arena_base > base_type
Definition: arena.h:277
void *__TBB_EXPORTED_FUNC NFS_Allocate(size_t n_element, size_t element_size, void *hint)
Allocate memory on cache/sector line boundary.
arena(market &, unsigned max_num_workers, unsigned num_reserved_slots)
Constructor.
Definition: arena.cpp:182
static int unsigned num_arena_slots(unsigned num_slots)
Definition: arena.h:292
const size_t NFS_MaxLineSize
Compile-time constant that is upper bound on cache line/sector size.
Definition: tbb_stddef.h:216
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
static int allocation_size(unsigned num_slots)
Definition: arena.h:296
size_t __TBB_EXPORTED_FUNC NFS_GetLineSize()
Cache/sector line size.

References __TBB_ASSERT, allocation_size(), arena(), tbb::internal::NFS_Allocate(), tbb::internal::NFS_GetLineSize(), tbb::internal::NFS_MaxLineSize, and num_arena_slots().

Referenced by tbb::internal::market::create_arena().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ allocation_size()

static int tbb::internal::arena::allocation_size ( unsigned  num_slots)
inlinestatic

Definition at line 296 of file arena.h.

296  {
297  return sizeof(base_type) + num_slots * (sizeof(mail_outbox) + sizeof(arena_slot));
298  }
padded< arena_base > base_type
Definition: arena.h:277

Referenced by allocate_arena(), and free_arena().

Here is the caller graph for this function:

◆ enqueue_task()

void tbb::internal::arena::enqueue_task ( task t,
intptr_t  prio,
FastRandom random 
)

enqueue a task into starvation-resistance queue

Definition at line 553 of file arena.cpp.

554 {
555 #if __TBB_RECYCLE_TO_ENQUEUE
556  __TBB_ASSERT( t.state()==task::allocated || t.state()==task::to_enqueue, "attempt to enqueue task with inappropriate state" );
557 #else
558  __TBB_ASSERT( t.state()==task::allocated, "attempt to enqueue task that is not in 'allocated' state" );
559 #endif
560  t.prefix().state = task::ready;
561  t.prefix().extra_state |= es_task_enqueued; // enqueued task marker
562 
563 #if TBB_USE_ASSERT
564  if( task* parent = t.parent() ) {
565  internal::reference_count ref_count = parent->prefix().ref_count;
566  __TBB_ASSERT( ref_count!=0, "attempt to enqueue task whose parent has a ref_count==0 (forgot to set_ref_count?)" );
567  __TBB_ASSERT( ref_count>0, "attempt to enqueue task whose parent has a ref_count<0" );
568  parent->prefix().extra_state |= es_ref_count_active;
569  }
570  __TBB_ASSERT(t.prefix().affinity==affinity_id(0), "affinity is ignored for enqueued tasks");
571 #endif /* TBB_USE_ASSERT */
572 #if __TBB_PREVIEW_CRITICAL_TASKS
573  if( prio == internal::priority_critical || internal::is_critical( t ) ) {
574  // TODO: consider using of 'scheduler::handled_as_critical'
576  generic_scheduler* s = governor::local_scheduler_if_initialized();
577  ITT_NOTIFY(sync_releasing, &my_critical_task_stream);
578  if( s && s->my_arena_slot ) {
579  // Scheduler is initialized and it is attached to the arena,
580  // propagate isolation level to critical task
581 #if __TBB_TASK_ISOLATION
582  t.prefix().isolation = s->my_innermost_running_task->prefix().isolation;
583 #endif
584  unsigned& lane = s->my_arena_slot->hint_for_critical;
585  my_critical_task_stream.push( &t, 0, tbb::internal::subsequent_lane_selector(lane) );
586  } else {
587  // Either scheduler is not initialized or it is not attached to the arena
588  // use random lane for the task
589  my_critical_task_stream.push( &t, 0, internal::random_lane_selector(random) );
590  }
591  advertise_new_work<work_spawned>();
592  return;
593  }
594 #endif /* __TBB_PREVIEW_CRITICAL_TASKS */
595 
597 #if __TBB_TASK_PRIORITY
598  intptr_t p = prio ? normalize_priority(priority_t(prio)) : normalized_normal_priority;
599  assert_priority_valid(p);
600 #if __TBB_PREVIEW_CRITICAL_TASKS && __TBB_CPF_BUILD
601  my_task_stream.push( &t, p, internal::random_lane_selector(random) );
602 #else
603  my_task_stream.push( &t, p, random );
604 #endif
605  if ( p != my_top_priority )
606  my_market->update_arena_priority( *this, p );
607 #else /* !__TBB_TASK_PRIORITY */
608  __TBB_ASSERT_EX(prio == 0, "the library is not configured to respect the task priority");
609 #if __TBB_PREVIEW_CRITICAL_TASKS && __TBB_CPF_BUILD
610  my_task_stream.push( &t, 0, internal::random_lane_selector(random) );
611 #else
612  my_task_stream.push( &t, 0, random );
613 #endif
614 #endif /* !__TBB_TASK_PRIORITY */
615  advertise_new_work<work_enqueued>();
616 #if __TBB_TASK_PRIORITY
617  if ( p != my_top_priority )
618  my_market->update_arena_priority( *this, p );
619 #endif /* __TBB_TASK_PRIORITY */
620 }
task_stream< num_priority_levels > my_task_stream
Task pool for the tasks scheduled via task::enqueue() method.
Definition: arena.h:168
task is in ready pool, or is going to be put there, or was just taken off.
Definition: task.h:630
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id parent
void make_critical(task &t)
Definition: task.h:1002
static const int priority_critical
Definition: task.h:302
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
void const char const char int ITT_FORMAT __itt_group_sync s
Set if ref_count might be changed by another thread. Used for debugging.
intptr_t reference_count
A reference count.
Definition: task.h:120
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
#define __TBB_ASSERT_EX(predicate, comment)
"Extended" version is useful to suppress warnings if a variable is only used with an assert
Definition: tbb_stddef.h:167
unsigned short affinity_id
An id as used for specifying affinity.
Definition: task.h:128
task object is freshly allocated or recycled.
Definition: task.h:632
market * my_market
The market that owns this arena.
Definition: arena.h:228
static generic_scheduler * local_scheduler_if_initialized()
Definition: governor.h:139
void push(task *source, int level, FastRandom &random)
Push a task into a lane.
Definition: task_stream.h:101
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:116
void const char const char int ITT_FORMAT __itt_group_sync p
priority_t
Definition: task.h:306
bool is_critical(task &t)
Definition: task.h:1003

References __TBB_ASSERT, __TBB_ASSERT_EX, tbb::internal::task_prefix::affinity, tbb::task::allocated, tbb::internal::es_ref_count_active, tbb::internal::es_task_enqueued, tbb::internal::task_prefix::extra_state, tbb::internal::is_critical(), tbb::internal::task_prefix::isolation, ITT_NOTIFY, tbb::internal::governor::local_scheduler_if_initialized(), tbb::internal::make_critical(), tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_task_stream, p, parent, tbb::task::parent(), tbb::task::prefix(), tbb::internal::priority_critical, tbb::internal::task_stream< Levels >::push(), tbb::task::ready, s, tbb::internal::task_prefix::state, tbb::task::state(), and sync_releasing.

Referenced by tbb::internal::custom_scheduler< SchedulerTraits >::tally_completion_of_predecessor().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ free_arena()

void tbb::internal::arena::free_arena ( )

Completes arena shutdown, destructs and deallocates it.

Definition at line 252 of file arena.cpp.

252  {
253  __TBB_ASSERT( is_alive(my_guard), NULL );
254  __TBB_ASSERT( !my_references, "There are threads in the dying arena" );
255  __TBB_ASSERT( !my_num_workers_requested && !my_num_workers_allotted, "Dying arena requests workers" );
256  __TBB_ASSERT( my_pool_state == SNAPSHOT_EMPTY || !my_max_num_workers, "Inconsistent state of a dying arena" );
257 #if __TBB_ENQUEUE_ENFORCED_CONCURRENCY
258  __TBB_ASSERT( !my_global_concurrency_mode, NULL );
259 #endif
260 #if !__TBB_STATISTICS_EARLY_DUMP
261  GATHER_STATISTIC( dump_arena_statistics() );
262 #endif
263  poison_value( my_guard );
264  intptr_t drained = 0;
265  for ( unsigned i = 0; i < my_num_slots; ++i ) {
266  __TBB_ASSERT( !my_slots[i].my_scheduler, "arena slot is not empty" );
267  // TODO: understand the assertion and modify
268  // __TBB_ASSERT( my_slots[i].task_pool == EmptyTaskPool, NULL );
269  __TBB_ASSERT( my_slots[i].head == my_slots[i].tail, NULL ); // TODO: replace by is_quiescent_local_task_pool_empty
271 #if __TBB_STATISTICS
272  NFS_Free( my_slots[i].my_counters );
273 #endif /* __TBB_STATISTICS */
274  drained += mailbox(i+1).drain();
275  }
276  __TBB_ASSERT( my_task_stream.drain()==0, "Not all enqueued tasks were executed");
277 #if __TBB_PREVIEW_RESUMABLE_TASKS
278  // Cleanup coroutines/schedulers cache
279  my_co_cache.cleanup();
280 #endif
281 #if __TBB_PREVIEW_CRITICAL_TASKS
282  __TBB_ASSERT( my_critical_task_stream.drain()==0, "Not all critical tasks were executed");
283 #endif
284 #if __TBB_COUNT_TASK_NODES
285  my_market->update_task_node_count( -drained );
286 #endif /* __TBB_COUNT_TASK_NODES */
287  // remove an internal reference
288  my_market->release( /*is_public=*/false, /*blocking_terminate=*/false );
289 #if __TBB_TASK_GROUP_CONTEXT
290  __TBB_ASSERT( my_default_ctx, "Master thread never entered the arena?" );
291  my_default_ctx->~task_group_context();
292  NFS_Free(my_default_ctx);
293 #endif /* __TBB_TASK_GROUP_CONTEXT */
294 #if __TBB_ARENA_OBSERVER
295  if ( !my_observers.empty() )
296  my_observers.clear();
297 #endif /* __TBB_ARENA_OBSERVER */
298  void* storage = &mailbox(my_num_slots);
299  __TBB_ASSERT( my_references == 0, NULL );
301  this->~arena();
302 #if TBB_USE_ASSERT > 1
303  memset( storage, 0, allocation_size(my_num_slots) );
304 #endif /* TBB_USE_ASSERT */
305  NFS_Free( storage );
306 }
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain ITT_FORMAT p const __itt_domain __itt_string_handle unsigned long long ITT_FORMAT lu const __itt_domain __itt_id __itt_string_handle __itt_metadata_type size_t void ITT_FORMAT p const __itt_domain __itt_id __itt_string_handle const wchar_t size_t ITT_FORMAT lu const __itt_domain __itt_id __itt_relation __itt_id tail
static const pool_state_t SNAPSHOT_EMPTY
No tasks to steal since last snapshot was taken.
Definition: arena.h:314
tbb::atomic< uintptr_t > my_pool_state
Current task pool state and estimate of available tasks amount.
Definition: arena.h:191
#define poison_value(g)
task_stream< num_priority_levels > my_task_stream
Task pool for the tasks scheduled via task::enqueue() method.
Definition: arena.h:168
bool release(bool is_public, bool blocking_terminate)
Decrements market's refcount and destroys it in the end.
Definition: market.cpp:175
unsigned my_num_slots
The number of slots in the arena.
Definition: arena.h:246
unsigned my_num_workers_allotted
The number of workers that have been marked out by the resource manager to service the arena.
Definition: arena.h:143
unsigned my_max_num_workers
The number of workers requested by the master thread owning the arena.
Definition: arena.h:181
atomic< unsigned > my_references
Reference counter for the arena.
Definition: arena.h:149
arena(market &, unsigned max_num_workers, unsigned num_reserved_slots)
Constructor.
Definition: arena.cpp:182
arena_slot my_slots[1]
Definition: arena.h:386
void __TBB_EXPORTED_FUNC NFS_Free(void *)
Free memory allocated by NFS_Allocate.
void free_task_pool()
Deallocate task pool that was allocated by means of allocate_task_pool.
market * my_market
The market that owns this arena.
Definition: arena.h:228
intptr_t drain()
Destroys all remaining tasks in every lane. Returns the number of destroyed tasks.
Definition: task_stream.h:145
mail_outbox & mailbox(affinity_id id)
Get reference to mailbox corresponding to given affinity_id.
Definition: arena.h:301
int my_num_workers_requested
The number of workers that are currently requested from the resource manager.
Definition: arena.h:184
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
static int allocation_size(unsigned num_slots)
Definition: arena.h:296
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain ITT_FORMAT p const __itt_domain __itt_string_handle unsigned long long ITT_FORMAT lu const __itt_domain __itt_id __itt_string_handle __itt_metadata_type size_t void ITT_FORMAT p const __itt_domain __itt_id __itt_string_handle const wchar_t size_t ITT_FORMAT lu const __itt_domain __itt_id head
intptr_t drain()
Drain the mailbox.
Definition: mailbox.h:168
#define GATHER_STATISTIC(x)

References __TBB_ASSERT, allocation_size(), tbb::internal::task_stream< Levels >::drain(), tbb::internal::mail_outbox::drain(), tbb::internal::arena_slot::free_task_pool(), GATHER_STATISTIC, head, mailbox(), tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_max_num_workers, tbb::internal::arena_base::my_num_slots, tbb::internal::arena_base::my_num_workers_allotted, tbb::internal::arena_base::my_num_workers_requested, tbb::internal::arena_base::my_pool_state, tbb::internal::arena_base::my_references, my_slots, tbb::internal::arena_base::my_task_stream, tbb::internal::NFS_Free(), poison_value, tbb::internal::market::release(), SNAPSHOT_EMPTY, and tail.

Referenced by tbb::internal::market::try_destroy_arena().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ has_enqueued_tasks()

bool tbb::internal::arena::has_enqueued_tasks ( )

Check for the presence of enqueued tasks at all priority levels.

Definition at line 382 of file arena.cpp.

382  {
383  // Look for enqueued tasks at all priority levels
384  for ( int p = 0; p < num_priority_levels; ++p )
385  if ( !my_task_stream.empty(p) )
386  return true;
387  return false;
388 }
task_stream< num_priority_levels > my_task_stream
Task pool for the tasks scheduled via task::enqueue() method.
Definition: arena.h:168
bool empty(int level)
Checks existence of a task.
Definition: task_stream.h:138
void const char const char int ITT_FORMAT __itt_group_sync p
static const intptr_t num_priority_levels

References tbb::internal::task_stream< Levels >::empty(), tbb::internal::arena_base::my_task_stream, tbb::internal::num_priority_levels, and p.

Referenced by restore_priority_if_need().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ is_busy_or_empty()

static bool tbb::internal::arena::is_busy_or_empty ( pool_state_t  s)
inlinestatic

No tasks to steal or snapshot is being taken.

Definition at line 327 of file arena.h.

327 { return s < SNAPSHOT_FULL; }
static const pool_state_t SNAPSHOT_FULL
At least one task has been offered for stealing since the last snapshot started.
Definition: arena.h:317
void const char const char int ITT_FORMAT __itt_group_sync s

References s, and SNAPSHOT_FULL.

Referenced by advertise_new_work().

Here is the caller graph for this function:

◆ is_out_of_work()

bool tbb::internal::arena::is_out_of_work ( )

Check if there is job anywhere in arena.

Return true if no job or if arena is being cleaned up.

Definition at line 410 of file arena.cpp.

410  {
411  // TODO: rework it to return at least a hint about where a task was found; better if the task itself.
412  for(;;) {
413  pool_state_t snapshot = my_pool_state;
414  switch( snapshot ) {
415  case SNAPSHOT_EMPTY:
416  return true;
417  case SNAPSHOT_FULL: {
418  // Use unique id for "busy" in order to avoid ABA problems.
419  const pool_state_t busy = pool_state_t(&busy);
420  // Request permission to take snapshot
421  if( my_pool_state.compare_and_swap( busy, SNAPSHOT_FULL )==SNAPSHOT_FULL ) {
422  // Got permission. Take the snapshot.
423  // NOTE: This is not a lock, as the state can be set to FULL at
424  // any moment by a thread that spawns/enqueues new task.
425  size_t n = my_limit;
426  // Make local copies of volatile parameters. Their change during
427  // snapshot taking procedure invalidates the attempt, and returns
428  // this thread into the dispatch loop.
429 #if __TBB_TASK_PRIORITY
430  uintptr_t reload_epoch = __TBB_load_with_acquire( my_reload_epoch );
431  intptr_t top_priority = my_top_priority;
432  // Inspect primary task pools first
433 #endif /* __TBB_TASK_PRIORITY */
434  size_t k;
435  for( k=0; k<n; ++k ) {
436  if( my_slots[k].task_pool != EmptyTaskPool &&
438  {
439  // k-th primary task pool is nonempty and does contain tasks.
440  break;
441  }
442  if( my_pool_state!=busy )
443  return false; // the work was published
444  }
445  __TBB_ASSERT( k <= n, NULL );
446  bool work_absent = k == n;
447 #if __TBB_PREVIEW_CRITICAL_TASKS
448  bool no_critical_tasks = my_critical_task_stream.empty(0);
449  work_absent &= no_critical_tasks;
450 #endif
451 #if __TBB_TASK_PRIORITY
452  // Variable tasks_present indicates presence of tasks at any priority
453  // level, while work_absent refers only to the current priority.
454  bool tasks_present = !work_absent || my_orphaned_tasks;
455  bool dequeuing_possible = false;
456  if ( work_absent ) {
457  // Check for the possibility that recent priority changes
458  // brought some tasks to the current priority level
459 
460  uintptr_t abandonment_epoch = my_abandonment_epoch;
461  // Master thread's scheduler needs special handling as it
462  // may be destroyed at any moment (workers' schedulers are
463  // guaranteed to be alive while at least one thread is in arena).
464  // The lock below excludes concurrency with task group state change
465  // propagation and guarantees lifetime of the master thread.
466  the_context_state_propagation_mutex.lock();
467  work_absent = !may_have_tasks( my_slots[0].my_scheduler, tasks_present, dequeuing_possible );
468  the_context_state_propagation_mutex.unlock();
469  // The following loop is subject to data races. While k-th slot's
470  // scheduler is being examined, corresponding worker can either
471  // leave to RML or migrate to another arena.
472  // But the races are not prevented because all of them are benign.
473  // First, the code relies on the fact that worker thread's scheduler
474  // object persists until the whole library is deinitialized.
475  // Second, in the worst case the races can only cause another
476  // round of stealing attempts to be undertaken. Introducing complex
477  // synchronization into this coldest part of the scheduler's control
478  // flow does not seem to make sense because it both is unlikely to
479  // ever have any observable performance effect, and will require
480  // additional synchronization code on the hotter paths.
481  for( k = 1; work_absent && k < n; ++k ) {
482  if( my_pool_state!=busy )
483  return false; // the work was published
484  work_absent = !may_have_tasks( my_slots[k].my_scheduler, tasks_present, dequeuing_possible );
485  }
486  // Preclude premature switching arena off because of a race in the previous loop.
487  work_absent = work_absent
488  && !__TBB_load_with_acquire(my_orphaned_tasks)
489  && abandonment_epoch == my_abandonment_epoch;
490  }
491 #endif /* __TBB_TASK_PRIORITY */
492  // Test and test-and-set.
493  if( my_pool_state==busy ) {
494 #if __TBB_TASK_PRIORITY
495  bool no_fifo_tasks = my_task_stream.empty(top_priority);
496  work_absent = work_absent && (!dequeuing_possible || no_fifo_tasks)
497  && top_priority == my_top_priority && reload_epoch == my_reload_epoch;
498 #else
499  bool no_fifo_tasks = my_task_stream.empty(0);
500  work_absent = work_absent && no_fifo_tasks;
501 #endif /* __TBB_TASK_PRIORITY */
502  if( work_absent ) {
503 #if __TBB_TASK_PRIORITY
504  if ( top_priority > my_bottom_priority ) {
505  if ( my_market->lower_arena_priority(*this, top_priority - 1, reload_epoch)
506  && !my_task_stream.empty(top_priority) )
507  {
508  atomic_update( my_skipped_fifo_priority, top_priority, std::less<intptr_t>());
509  }
510  }
511  else if ( !tasks_present && !my_orphaned_tasks && no_fifo_tasks ) {
512 #endif /* __TBB_TASK_PRIORITY */
513  // save current demand value before setting SNAPSHOT_EMPTY,
514  // to avoid race with advertise_new_work.
515  int current_demand = (int)my_max_num_workers;
516  if( my_pool_state.compare_and_swap( SNAPSHOT_EMPTY, busy )==busy ) {
517  // This thread transitioned pool to empty state, and thus is
518  // responsible for telling the market that there is no work to do.
519  my_market->adjust_demand( *this, -current_demand );
521  return true;
522  }
523  return false;
524 #if __TBB_TASK_PRIORITY
525  }
526 #endif /* __TBB_TASK_PRIORITY */
527  }
528  // Undo previous transition SNAPSHOT_FULL-->busy, unless another thread undid it.
529  my_pool_state.compare_and_swap( SNAPSHOT_FULL, busy );
530  }
531  }
532  return false;
533  }
534  default:
535  // Another thread is taking a snapshot.
536  return false;
537  }
538  }
539 }
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain ITT_FORMAT p const __itt_domain __itt_string_handle unsigned long long ITT_FORMAT lu const __itt_domain __itt_id __itt_string_handle __itt_metadata_type size_t void ITT_FORMAT p const __itt_domain __itt_id __itt_string_handle const wchar_t size_t ITT_FORMAT lu const __itt_domain __itt_id __itt_relation __itt_id tail
static const pool_state_t SNAPSHOT_EMPTY
No tasks to steal since last snapshot was taken.
Definition: arena.h:314
tbb::atomic< uintptr_t > my_pool_state
Current task pool state and estimate of available tasks amount.
Definition: arena.h:191
T1 atomic_update(tbb::atomic< T1 > &dst, T2 newValue, Pred compare)
Atomically replaces value of dst with newValue if they satisfy condition of compare predicate.
Definition: tbb_misc.h:191
task_stream< num_priority_levels > my_task_stream
Task pool for the tasks scheduled via task::enqueue() method.
Definition: arena.h:168
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:735
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain ITT_FORMAT p const __itt_domain __itt_string_handle unsigned long long ITT_FORMAT lu const __itt_domain __itt_id __itt_string_handle __itt_metadata_type size_t void ITT_FORMAT p const __itt_domain __itt_id __itt_string_handle const wchar_t size_t ITT_FORMAT lu const __itt_domain __itt_id __itt_relation __itt_id ITT_FORMAT p const wchar_t int ITT_FORMAT __itt_group_mark d int
void restore_priority_if_need()
If enqueued tasks found, restore arena priority and task presence status.
Definition: arena.cpp:390
uintptr_t pool_state_t
Definition: arena.h:311
static const pool_state_t SNAPSHOT_FULL
At least one task has been offered for stealing since the last snapshot started.
Definition: arena.h:317
unsigned my_max_num_workers
The number of workers requested by the master thread owning the arena.
Definition: arena.h:181
void adjust_demand(arena &, int delta)
Request that arena's need in workers should be adjusted.
Definition: market.cpp:556
T __TBB_load_with_acquire(const volatile T &location)
Definition: tbb_machine.h:709
bool empty(int level)
Checks existence of a task.
Definition: task_stream.h:138
arena_slot my_slots[1]
Definition: arena.h:386
market * my_market
The market that owns this arena.
Definition: arena.h:228
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
atomic< unsigned > my_limit
The maximal number of currently busy slots.
Definition: arena.h:157
#define EmptyTaskPool
Definition: scheduler.h:46
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain ITT_FORMAT p const __itt_domain __itt_string_handle unsigned long long ITT_FORMAT lu const __itt_domain __itt_id __itt_string_handle __itt_metadata_type size_t void ITT_FORMAT p const __itt_domain __itt_id __itt_string_handle const wchar_t size_t ITT_FORMAT lu const __itt_domain __itt_id head

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_load_with_acquire(), tbb::internal::market::adjust_demand(), tbb::internal::atomic_update(), tbb::internal::task_stream< Levels >::empty(), EmptyTaskPool, head, int, tbb::internal::arena_base::my_limit, tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_max_num_workers, tbb::internal::arena_base::my_pool_state, my_slots, tbb::internal::arena_base::my_task_stream, restore_priority_if_need(), SNAPSHOT_EMPTY, SNAPSHOT_FULL, and tail.

Referenced by on_thread_leaving().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ is_recall_requested()

bool tbb::internal::arena::is_recall_requested ( ) const
inline

Check if the recall is requested by the market.

Definition at line 335 of file arena.h.

335  {
337  }
unsigned my_num_workers_allotted
The number of workers that have been marked out by the resource manager to service the arena.
Definition: arena.h:143
unsigned num_workers_active() const
The number of workers active in the arena.
Definition: arena.h:330

References tbb::internal::arena_base::my_num_workers_allotted, and num_workers_active().

Referenced by process().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ mailbox()

mail_outbox& tbb::internal::arena::mailbox ( affinity_id  id)
inline

Get reference to mailbox corresponding to given affinity_id.

Definition at line 301 of file arena.h.

301  {
302  __TBB_ASSERT( 0<id, "affinity id must be positive integer" );
303  __TBB_ASSERT( id <= my_num_slots, "affinity id out of bounds" );
304 
305  return ((mail_outbox*)this)[-(int)id];
306  }
unsigned my_num_slots
The number of slots in the arena.
Definition: arena.h:246
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain ITT_FORMAT p const __itt_domain __itt_string_handle unsigned long long ITT_FORMAT lu const __itt_domain __itt_id __itt_string_handle __itt_metadata_type size_t void ITT_FORMAT p const __itt_domain __itt_id __itt_string_handle const wchar_t size_t ITT_FORMAT lu const __itt_domain __itt_id __itt_relation __itt_id ITT_FORMAT p const wchar_t int ITT_FORMAT __itt_group_mark d int
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165

References __TBB_ASSERT, int, and tbb::internal::arena_base::my_num_slots.

Referenced by arena(), tbb::internal::generic_scheduler::attach_mailbox(), free_arena(), and tbb::internal::generic_scheduler::prepare_for_spawning().

Here is the caller graph for this function:

◆ num_arena_slots()

static int unsigned tbb::internal::arena::num_arena_slots ( unsigned  num_slots)
inlinestatic

Definition at line 292 of file arena.h.

292  {
293  return max(2u, num_slots);
294  }
T max(const T &val1, const T &val2)
Utility template function returning greater of the two values.
Definition: tbb_misc.h:124

References tbb::internal::max().

Referenced by allocate_arena(), and arena().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ num_workers_active()

unsigned tbb::internal::arena::num_workers_active ( ) const
inline

The number of workers active in the arena.

Definition at line 330 of file arena.h.

330  {
332  }
atomic< unsigned > my_references
Reference counter for the arena.
Definition: arena.h:149
static const unsigned ref_external_bits
The number of least significant bits for external references.
Definition: arena.h:320

References tbb::internal::arena_base::my_references, and ref_external_bits.

Referenced by tbb::internal::market::arena_in_need(), and is_recall_requested().

Here is the caller graph for this function:

◆ occupy_free_slot()

template<bool as_worker>
size_t tbb::internal::arena::occupy_free_slot ( generic_scheduler s)

Tries to occupy a slot in the arena. On success, returns the slot index; if no slot is available, returns out_of_arena.

Definition at line 86 of file arena.cpp.

86  {
87  // Firstly, masters try to occupy reserved slots
88  size_t index = as_worker ? out_of_arena : occupy_free_slot_in_range( s, 0, my_num_reserved_slots );
89  if ( index == out_of_arena ) {
90  // Secondly, all threads try to occupy all non-reserved slots
92  // Likely this arena is already saturated
93  if ( index == out_of_arena )
94  return out_of_arena;
95  }
96 
97  ITT_NOTIFY(sync_acquired, my_slots + index);
98  atomic_update( my_limit, (unsigned)(index + 1), std::less<unsigned>() );
99  return index;
100 }
T1 atomic_update(tbb::atomic< T1 > &dst, T2 newValue, Pred compare)
Atomically replaces value of dst with newValue if they satisfy condition of compare predicate.
Definition: tbb_misc.h:191
unsigned my_num_slots
The number of slots in the arena.
Definition: arena.h:246
static const size_t out_of_arena
Definition: arena.h:378
void const char const char int ITT_FORMAT __itt_group_sync s
arena_slot my_slots[1]
Definition: arena.h:386
size_t occupy_free_slot_in_range(generic_scheduler &s, size_t lower, size_t upper)
Tries to occupy a slot in the specified range.
Definition: arena.cpp:71
unsigned my_num_reserved_slots
The number of reserved slots (can be occupied only by masters).
Definition: arena.h:249
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:116
atomic< unsigned > my_limit
The maximal number of currently busy slots.
Definition: arena.h:157

References tbb::internal::atomic_update(), ITT_NOTIFY, tbb::internal::arena_base::my_limit, tbb::internal::arena_base::my_num_reserved_slots, tbb::internal::arena_base::my_num_slots, my_slots, occupy_free_slot_in_range(), out_of_arena, and s.

Referenced by process().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ occupy_free_slot_in_range()

size_t tbb::internal::arena::occupy_free_slot_in_range ( generic_scheduler s,
size_t  lower,
size_t  upper 
)

Tries to occupy a slot in the specified range.

Definition at line 71 of file arena.cpp.

71  {
72  if ( lower >= upper ) return out_of_arena;
73  // Start search for an empty slot from the one we occupied the last time
74  size_t index = s.my_arena_index;
75  if ( index < lower || index >= upper ) index = s.my_random.get() % (upper - lower) + lower;
76  __TBB_ASSERT( index >= lower && index < upper, NULL );
77  // Find a free slot
78  for ( size_t i = index; i < upper; ++i )
79  if ( occupy_slot(my_slots[i].my_scheduler, s) ) return i;
80  for ( size_t i = lower; i < index; ++i )
81  if ( occupy_slot(my_slots[i].my_scheduler, s) ) return i;
82  return out_of_arena;
83 }
static bool occupy_slot(generic_scheduler *&slot, generic_scheduler &s)
Definition: arena.cpp:67
static const size_t out_of_arena
Definition: arena.h:378
void const char const char int ITT_FORMAT __itt_group_sync s
arena_slot my_slots[1]
Definition: arena.h:386
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165

References __TBB_ASSERT, my_slots, tbb::internal::occupy_slot(), out_of_arena, and s.

Referenced by occupy_free_slot().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ on_thread_leaving()

template<unsigned ref_param>
void tbb::internal::arena::on_thread_leaving ( )
inline

Notification that worker or master leaves its arena.

Definition at line 390 of file arena.h.

390  {
391  //
392  // Implementation of arena destruction synchronization logic contained various
393  // bugs/flaws at the different stages of its evolution, so below is a detailed
394  // description of the issues taken into consideration in the framework of the
395  // current design.
396  //
397  // In case of using fire-and-forget tasks (scheduled via task::enqueue())
398  // master thread is allowed to leave its arena before all its work is executed,
399  // and market may temporarily revoke all workers from this arena. Since revoked
400  // workers never attempt to reset arena state to EMPTY and cancel its request
401  // to RML for threads, the arena object is destroyed only when both the last
402  // thread is leaving it and arena's state is EMPTY (that is its master thread
403  // left and it does not contain any work).
404  // Thus resetting arena to EMPTY state (as earlier TBB versions did) should not
405  // be done here (or anywhere else in the master thread to that matter); doing so
406  // can result either in arena's premature destruction (at least without
407  // additional costly checks in workers) or in unnecessary arena state changes
408  // (and ensuing workers migration).
409  //
410  // A worker that checks for work presence and transitions arena to the EMPTY
411  // state (in snapshot taking procedure arena::is_out_of_work()) updates
412  // arena::my_pool_state first and only then arena::my_num_workers_requested.
413  // So the check for work absence must be done against the latter field.
414  //
415  // In a time window between decrementing the active threads count and checking
416  // if there is an outstanding request for workers. New worker thread may arrive,
417  // finish remaining work, set arena state to empty, and leave decrementing its
418  // refcount and destroying. Then the current thread will destroy the arena
419  // the second time. To preclude it a local copy of the outstanding request
420  // value can be stored before decrementing active threads count.
421  //
422  // But this technique may cause two other problem. When the stored request is
423  // zero, it is possible that arena still has threads and they can generate new
424  // tasks and thus re-establish non-zero requests. Then all the threads can be
425  // revoked (as described above) leaving this thread the last one, and causing
426  // it to destroy non-empty arena.
427  //
428  // The other problem takes place when the stored request is non-zero. Another
429  // thread may complete the work, set arena state to empty, and leave without
430  // arena destruction before this thread decrements the refcount. This thread
431  // cannot destroy the arena either. Thus the arena may be "orphaned".
432  //
433  // In both cases we cannot dereference arena pointer after the refcount is
434  // decremented, as our arena may already be destroyed.
435  //
436  // If this is the master thread, the market is protected by refcount to it.
437  // In case of workers market's liveness is ensured by the RML connection
438  // rundown protocol, according to which the client (i.e. the market) lives
439  // until RML server notifies it about connection termination, and this
440  // notification is fired only after all workers return into RML.
441  //
442  // Thus if we decremented refcount to zero we ask the market to check arena
443  // state (including the fact if it is alive) under the lock.
444  //
445  uintptr_t aba_epoch = my_aba_epoch;
446  market* m = my_market;
447  __TBB_ASSERT(my_references >= ref_param, "broken arena reference counter");
448 #if __TBB_STATISTICS_EARLY_DUMP
449  // While still holding a reference to the arena, compute how many external references are left.
450  // If just one, dump statistics.
451  if ( modulo_power_of_two(my_references,ref_worker)==ref_param ) // may only be true with ref_external
452  GATHER_STATISTIC( dump_arena_statistics() );
453 #endif
454 #if __TBB_ENQUEUE_ENFORCED_CONCURRENCY
455  // When there is no workers someone must free arena, as
456  // without workers, no one calls is_out_of_work().
457  // Skip workerless arenas because they have no demand for workers.
458  // TODO: consider more strict conditions for the cleanup,
459  // because it can create the demand of workers,
460  // but the arena can be already empty (and so ready for destroying)
461  // TODO: Fix the race: while we check soft limit and it might be changed.
462  if( ref_param==ref_external && my_num_slots != my_num_reserved_slots
463  && 0 == m->my_num_workers_soft_limit && !my_global_concurrency_mode ) {
464  bool is_out = false;
465  for (int i=0; i<num_priority_levels; i++) {
466  is_out = is_out_of_work();
467  if (is_out)
468  break;
469  }
470  // We expect, that in worst case it's enough to have num_priority_levels-1
471  // calls to restore priorities and yet another is_out_of_work() to conform
472  // that no work was found. But as market::set_active_num_workers() can be called
473  // concurrently, can't guarantee last is_out_of_work() return true.
474  }
475 #endif
476  if ( (my_references -= ref_param ) == 0 )
477  m->try_destroy_arena( this, aba_epoch );
478 }
static const unsigned ref_external
Reference increment values for externals and workers.
Definition: arena.h:323
uintptr_t my_aba_epoch
ABA prevention marker.
Definition: arena.h:231
unsigned my_num_slots
The number of slots in the arena.
Definition: arena.h:246
bool is_out_of_work()
Check if there is job anywhere in arena.
Definition: arena.cpp:410
atomic< unsigned > my_references
Reference counter for the arena.
Definition: arena.h:149
unsigned my_num_reserved_slots
The number of reserved slots (can be occupied only by masters).
Definition: arena.h:249
argument_integer_type modulo_power_of_two(argument_integer_type arg, divisor_integer_type divisor)
A function to compute arg modulo divisor where divisor is a power of 2.
Definition: tbb_stddef.h:382
static const unsigned ref_worker
Definition: arena.h:324
market * my_market
The market that owns this arena.
Definition: arena.h:228
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
static const intptr_t num_priority_levels
#define GATHER_STATISTIC(x)

References __TBB_ASSERT, GATHER_STATISTIC, is_out_of_work(), tbb::internal::modulo_power_of_two(), tbb::internal::arena_base::my_aba_epoch, tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_num_reserved_slots, tbb::internal::arena_base::my_num_slots, tbb::internal::market::my_num_workers_soft_limit, tbb::internal::arena_base::my_references, tbb::internal::num_priority_levels, ref_external, ref_worker, and tbb::internal::market::try_destroy_arena().

Referenced by tbb::internal::generic_scheduler::cleanup_master().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ process()

void tbb::internal::arena::process ( generic_scheduler s)

Registers the worker with the arena and enters TBB scheduler dispatch loop.

Definition at line 102 of file arena.cpp.

102  {
103  __TBB_ASSERT( is_alive(my_guard), NULL );
104  __TBB_ASSERT( governor::is_set(&s), NULL );
105  __TBB_ASSERT( s.my_innermost_running_task == s.my_dummy_task, NULL );
106  __TBB_ASSERT( s.worker_outermost_level(), NULL );
107 
108  __TBB_ASSERT( my_num_slots > 1, NULL );
109 
110  size_t index = occupy_free_slot</*as_worker*/true>( s );
111  if ( index == out_of_arena )
112  goto quit;
113 
114  __TBB_ASSERT( index >= my_num_reserved_slots, "Workers cannot occupy reserved slots" );
115  s.attach_arena( this, index, /*is_master*/false );
116 
117 #if !__TBB_FP_CONTEXT
119 #endif
120 
121 #if __TBB_ARENA_OBSERVER
122  __TBB_ASSERT( !s.my_last_local_observer, "There cannot be notified local observers when entering arena" );
123  my_observers.notify_entry_observers( s.my_last_local_observer, /*worker=*/true );
124 #endif /* __TBB_ARENA_OBSERVER */
125 
126  // Task pool can be marked as non-empty if the worker occupies the slot left by a master.
127  if ( s.my_arena_slot->task_pool != EmptyTaskPool ) {
128  __TBB_ASSERT( s.my_inbox.is_idle_state(false), NULL );
129  s.local_wait_for_all( *s.my_dummy_task, NULL );
130  __TBB_ASSERT( s.my_inbox.is_idle_state(true), NULL );
131  }
132 
133  for ( ;; ) {
134  __TBB_ASSERT( s.my_innermost_running_task == s.my_dummy_task, NULL );
135  __TBB_ASSERT( s.worker_outermost_level(), NULL );
136  __TBB_ASSERT( is_alive(my_guard), NULL );
137  __TBB_ASSERT( s.is_quiescent_local_task_pool_reset(),
138  "Worker cannot leave arena while its task pool is not reset" );
139  __TBB_ASSERT( s.my_arena_slot->task_pool == EmptyTaskPool, "Empty task pool is not marked appropriately" );
140  // This check prevents relinquishing more than necessary workers because
141  // of the non-atomicity of the decision making procedure
142  if ( is_recall_requested() )
143  break;
144  // Try to steal a task.
145  // Passing reference count is technically unnecessary in this context,
146  // but omitting it here would add checks inside the function.
147  task* t = s.receive_or_steal_task( __TBB_ISOLATION_ARG( s.my_dummy_task->prefix().ref_count, no_isolation ) );
148  if (t) {
149  // A side effect of receive_or_steal_task is that my_innermost_running_task can be set.
150  // But for the outermost dispatch loop it has to be a dummy task.
151  s.my_innermost_running_task = s.my_dummy_task;
152  s.local_wait_for_all(*s.my_dummy_task,t);
153  }
154  }
155 #if __TBB_ARENA_OBSERVER
156  my_observers.notify_exit_observers( s.my_last_local_observer, /*worker=*/true );
157  s.my_last_local_observer = NULL;
158 #endif /* __TBB_ARENA_OBSERVER */
159 #if __TBB_TASK_PRIORITY
160  if ( s.my_offloaded_tasks )
161  orphan_offloaded_tasks( s );
162 #endif /* __TBB_TASK_PRIORITY */
163 #if __TBB_STATISTICS
164  ++s.my_counters.arena_roundtrips;
165  *my_slots[index].my_counters += s.my_counters;
166  s.my_counters.reset();
167 #endif /* __TBB_STATISTICS */
168  __TBB_store_with_release( my_slots[index].my_scheduler, (generic_scheduler*)NULL );
169  s.my_arena_slot = 0; // detached from slot
170  s.my_inbox.detach();
171  __TBB_ASSERT( s.my_inbox.is_idle_state(true), NULL );
172  __TBB_ASSERT( s.my_innermost_running_task == s.my_dummy_task, NULL );
173  __TBB_ASSERT( s.worker_outermost_level(), NULL );
174  __TBB_ASSERT( is_alive(my_guard), NULL );
175 quit:
176  // In contrast to earlier versions of TBB (before 3.0 U5) now it is possible
177  // that arena may be temporarily left unpopulated by threads. See comments in
178  // arena::on_thread_leaving() for more details.
179  on_thread_leaving<ref_worker>();
180 }
bool is_recall_requested() const
Check if the recall is requested by the market.
Definition: arena.h:335
unsigned my_num_slots
The number of slots in the arena.
Definition: arena.h:246
cpu_ctl_env my_cpu_ctl_env
FPU control settings of arena's master thread captured at the moment of arena instantiation.
Definition: arena.h:235
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:713
static bool is_set(generic_scheduler *s)
Used to check validity of the local scheduler TLS contents.
Definition: governor.cpp:120
static const size_t out_of_arena
Definition: arena.h:378
void const char const char int ITT_FORMAT __itt_group_sync s
arena_slot my_slots[1]
Definition: arena.h:386
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
#define __TBB_ISOLATION_ARG(arg1, isolation)
unsigned my_num_reserved_slots
The number of reserved slots (can be occupied only by masters).
Definition: arena.h:249
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:165
size_t occupy_free_slot(generic_scheduler &s)
Tries to occupy a slot in the arena. On success, returns the slot index; if no slot is available,...
Definition: arena.cpp:86
const isolation_tag no_isolation
Definition: task.h:133
#define EmptyTaskPool
Definition: scheduler.h:46

References __TBB_ASSERT, __TBB_ISOLATION_ARG, tbb::internal::__TBB_store_with_release(), EmptyTaskPool, is_recall_requested(), tbb::internal::governor::is_set(), tbb::internal::arena_base::my_cpu_ctl_env, tbb::internal::arena_base::my_num_reserved_slots, tbb::internal::arena_base::my_num_slots, my_slots, tbb::internal::no_isolation, occupy_free_slot(), out_of_arena, s, and tbb::internal::cpu_ctl_env::set_env().

Referenced by tbb::internal::market::process().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ restore_priority_if_need()

void tbb::internal::arena::restore_priority_if_need ( )
private

If enqueued tasks found, restore arena priority and task presence status.

Definition at line 390 of file arena.cpp.

390  {
391  // Check for the presence of enqueued tasks "lost" on some of
392  // priority levels because updating arena priority and switching
393  // arena into "populated" (FULL) state happen non-atomically.
394  // Imposing atomicity would require task::enqueue() to use a lock,
395  // which is unacceptable.
396  if ( has_enqueued_tasks() ) {
397  advertise_new_work<work_enqueued>();
398 #if __TBB_TASK_PRIORITY
399  // update_arena_priority() expects non-zero arena::my_num_workers_requested,
400  // so must be called after advertise_new_work<work_enqueued>()
401  for ( int p = 0; p < num_priority_levels; ++p )
402  if ( !my_task_stream.empty(p) ) {
403  if ( p < my_bottom_priority || p > my_top_priority )
404  my_market->update_arena_priority(*this, p);
405  }
406 #endif
407  }
408 }
task_stream< num_priority_levels > my_task_stream
Task pool for the tasks scheduled via task::enqueue() method.
Definition: arena.h:168
bool has_enqueued_tasks()
Check for the presence of enqueued tasks at all priority levels.
Definition: arena.cpp:382
bool empty(int level)
Checks existence of a task.
Definition: task_stream.h:138
market * my_market
The market that owns this arena.
Definition: arena.h:228
void const char const char int ITT_FORMAT __itt_group_sync p
static const intptr_t num_priority_levels

References tbb::internal::task_stream< Levels >::empty(), has_enqueued_tasks(), tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_task_stream, tbb::internal::num_priority_levels, and p.

Referenced by is_out_of_work().

Here is the call graph for this function:
Here is the caller graph for this function:

Member Data Documentation

◆ my_slots

◆ out_of_arena

const size_t tbb::internal::arena::out_of_arena = ~size_t(0)
static

Definition at line 378 of file arena.h.

Referenced by occupy_free_slot(), occupy_free_slot_in_range(), and process().

◆ ref_external

const unsigned tbb::internal::arena::ref_external = 1
static

Reference increment values for externals and workers.

Definition at line 323 of file arena.h.

Referenced by arena(), tbb::internal::generic_scheduler::cleanup_master(), and on_thread_leaving().

◆ ref_external_bits

const unsigned tbb::internal::arena::ref_external_bits = 12
static

The number of least significant bits for external references.

Definition at line 320 of file arena.h.

Referenced by num_workers_active().

◆ ref_worker

const unsigned tbb::internal::arena::ref_worker = 1<<ref_external_bits
static

Definition at line 324 of file arena.h.

Referenced by tbb::internal::market::arena_in_need(), and on_thread_leaving().

◆ SNAPSHOT_EMPTY

const pool_state_t tbb::internal::arena::SNAPSHOT_EMPTY = 0
static

No tasks to steal since last snapshot was taken.

Definition at line 314 of file arena.h.

Referenced by advertise_new_work(), free_arena(), is_out_of_work(), tbb::internal::market::try_destroy_arena(), and tbb::internal::generic_scheduler::wait_until_empty().

◆ SNAPSHOT_FULL

const pool_state_t tbb::internal::arena::SNAPSHOT_FULL = pool_state_t(-1)
static

At least one task has been offered for stealing since the last snapshot started.

Definition at line 317 of file arena.h.

Referenced by advertise_new_work(), is_busy_or_empty(), and is_out_of_work().


The documentation for this class was generated from the following files:

Copyright © 2005-2019 Intel Corporation. All Rights Reserved.

Intel, Pentium, Intel Xeon, Itanium, Intel XScale and VTune are registered trademarks or trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

* Other names and brands may be claimed as the property of others.